:lang: en
:toc:

[[cha:remap]]
= Remap Extending G-code

:ini: {basebackend@docbook:'':ini}
:hal: {basebackend@docbook:'':hal}
:ngc: {basebackend@docbook:'':ngc}

== Introduction: Extending the RS274NGC Interpreter by Remapping Codes

=== A Definition: Remapping Codes

By 'remapping codes' we mean one of the following:

. Define the semantics of new - that is, currently unallocated - M- or G-codes
. Redefine the semantics of a - currently limited - set of existing codes.

=== Why would you want to extend the RS274NGC Interpreter?

The set of codes (M,G,T,S,F) currently understood by the RS274NGC
interpreter is fixed and cannot be extended by configuration options.

In particular, some of these codes implement a fixed sequence of steps
to be executed. While some of these, like M6, can be moderately
configured by activating or skipping some of these steps through INI
file options, overall the behavior is fairly rigid. So - if your
are happy with this situation, then this manual section is not for you.

In many cases, this means that supporting a non 'out of the box'
configuration or machine is either cumbersome or impossible, or
requires resorting to changes at the 'C/C\+\+' language level. The latter
is unpopular for good reasons - changing internals requires in-depth
understanding of interpreter internals, and moreover brings its own
set of support issues. While it is conceivable that certain patches
might make their way into the main LinuxCNC distribution, the result of
this approach is a hodge-podge of special-case solutions.

A good example for this deficiency is tool change support in LinuxCNC:
While random tool changers are well supported, it is next to impossible
to reasonably define a configuration for a manual-tool change machine
with, for example, an automatic tool length offset switch being
visited after a tool change, and offsets set accordingly. Also, while a
patch for a very specific rack tool changer exists, it has not found
its way back into the main code base.

However, many of these things may be fixed by using an O-word
procedure instead of a built in code - whenever the - insufficient -
built in code is to be executed, call the O-word procedure
instead. While possible, it is cumbersome - it requires source-editing
of NGC programs, replacing all calls to the deficient code by a an
O-word procedure call.

In its simplest form, a remapped code isn't much more than a
spontaneous call to an O-word procedure. This happens behind the scenes -
the procedure is visible at the configuration level, but not at the
NGC program level.

Generally, the behavior of a remapped code may be defined in the following ways:

- You define a O-word subroutine which implements the desired behavior
- Alternatively, you may employ a Python function which extends the interpreter's behavior.

.How to glue things together
M- and G-codes, and O-words subroutine calls have some fairly different syntax.

O-word procedures, for example, take positional parameters
with a specific syntax like so:

[source,{ngc}]
---------------------------------------------------------------------
o<test> call [1.234] [4.65]
---------------------------------------------------------------------

whereas M- or G-codes typically take required or optional 'word'
parameters. For instance, G76 (threading) requires the P,Z,I,J and K
words, and optionally takes the R,Q,H, E and L words.

So it isn't simply enough to say 'whenever you encounter code X, please
call procedure Y' - at least some checking and conversion of parameters
needs to happen. This calls for some 'glue code' between the new code,
and its corresponding NGC procedure to execute before passing control
to the NGC procedure.

This glue code is impossible to write as an O-word procedure itself
since the RS274NGC language lacks the introspective capabilities and
access into interpreter internal data structures to achieve the
required effect. Doing the glue code in - again - 'C/C\+\+' would be an
inflexible and therefore unsatisfactory solution.

.How Embedded Python fits in

To make a simple situation easy and a complex situation solvable, the
glue issue is addressed as follows:

- for simple situations, a built-in glue procedure (`argspec`) covers most
  common parameter passing requirements
- for remapping T,M6,M61,S,F there is some standard Python glue which should
  cover most situations, see <<remap:standard-glue,Standard Glue>>
- for more complex situations, one can write your own Python glue to implement
  new behavior.

Embedded Python functions in the Interpreter started out as glue code,
but turned out very useful well beyond that. Users familiar with
Python will likely find it easier to write remapped codes, glue,
O-word procedures, etc. in pure Python, without resorting to the
somewhat cumbersome RS274NGC language at all.

.A Word on Embedded Python
Many people are familiar with 'extending' the Python interpreter by
'C/C\+\+' modules, and this is heavily used in LinuxCNC to access Task,
HAL and Interpreter internals from Python scripts. 'Extending
Python' basically means: your Python script executes as 'it is in the
driver seat', and may access non-Python code by importing and using
extension modules written in 'C/C\+\+'. Examples for this are the LinuxCNC
`hal`, `gcode` and `emc` modules.

Embedded Python is a bit different and less commonly known: The
main program is written in C/C++ and may use Python like a
subroutine. This is powerful extension mechanism and the basis for the
'scripting extensions' found in many successful software
packages. Embedded Python code may access 'C/C\+\+' variables and
functions through a similar extension module method.

[[remap:getting-started]]
==  Getting started

Defining a code involves the following steps:

- Pick a code - either use an unallocated code, or redefine an existing code.
- Decide how parameters are handled.
- Decide if and how results are handled.
- Decide about the execution sequence.

=== Builtin Remaps

Please note that currently only some existing codes can be redefined,
while there are many 'free' codes that may be available for
remapping. When developing redefined existing code, it is a good idea to
start with an unassigned G- or M- code, so that you can use both
an existing behavior as well as a new one. When you're done, redefine
the existing code to use your remapping configuration.

- The current set of unused M-codes, available for user definition, can
  be found in the <<remap:unallocated-m-codes,unallocated M-codes section>>,
- For G-codes, see the <<remap:unallocated-g-codes,unallocated G-codes list>>.
- Existing codes that can be reassigned are listed in the
  <<remap:remappable-codes,remappable codes section>>.

There are currently two complete Python-only remaps that are available
in stdglue.py:

* ignore_m6
* index_lathe_tool_with_wear

These are meant for use with lathe.
Lathes don't use M6 to index the tools, they use the T command.

This remap also adds wear offsets to the tool offset, ie. T201 would
index to tool 2 (with tool 2's tool offset) and adds wear offset 1.
In the tool table, tools numbers above 10000 are wear offsets, ie in the
tool table, tool 10001 would be wear offset 1.

Here is what you need in the INI to use them:

[source,{ini}]
----
[RS274NGC]
REMAP=T python=index_lathe_tool_with_wear
REMAP=M6 python=ignore_m6

[PYTHON]
# where to find the Python code:

# code specific for this configuration
PATH_PREPEND=./

# generic support code - make sure this actually points to Python-stdglue
PATH_APPEND=../../nc_files/remap_lib/python-stdglue/

# import the following Python module
TOPLEVEL=toplevel.py

# the higher the more verbose tracing of the Python plugin
LOG_LEVEL = 0
----

You must also add the required Python file in your configuration folder.

<<remap:upgrading-an-existing,Upgrade an existing configuration>>

=== Picking a code

Note that currently only a few existing codes may be redefined,
whereas there are many 'free' codes which might be made available by
remapping. When developing a redefined existing code, it might be a
good idea to start with an unallocated G- or M-code so both the
existing and new behavior can be exercised. When done, redefine the
existing code to use your remapping setup.

- the current set of unused M-codes open to user definition can be found
  <<remap:unallocated-m-codes,here>>,
- unallocated G-codes are listed <<remap:unallocated-g-codes,here>>.
- Existing codes which may be remapped are listed <<remap:remappable-codes,here>>.

[[remap:parameter-handling]]
===  Parameter handling

Let's assume the new code will be defined by an NGC procedure, and needs
some parameters, some of which might be required, others might be
optional. We have the following options to feed values to the
procedure:

// . <<remap:extracting-words,extracting words from the current block>>
. extracting words from the current block  and pass them to the
  procedure as parameters (like `X22.34` or `P47`)
. referring to <<gcode:ini-hal-params,INI file variables>>
. referring to global variables (like `#2200 = 47.11` or
  `#<_global_param> = 315.2`)

The first method is preferred for parameters of dynamic nature, like
positions. You need to define which words on the current block have
any meaning for your new code, and specify how that is passed to the
NGC procedure. Any easy way is to use the
<<sub:argspec-parameter,argspec statement>>. A custom prolog might
provide better error messages.

Using to INI file variables is most useful for referring to setup
information for your machine, for instance a fixed position like a
tool-length sensor position. The advantage of this method is that the
parameters are fixed for your configuration regardless which NGC file
you're currently executing.

Referring to global variables is always possible, but they are easily
overlooked.

Note there's a limited supply of words which may be used as
parameters, so one might need to fall back to the second and third
methods if many parameters are needed.

[[remap:handling-results]]
=== Handling results

Your new code might succeed or fail, for instance if passed an invalid
parameter combination. Or you might choose to 'just execute' the
procedure and disregard results, in which case there isn't much work to do.

Epilog handlers help in processing results of remap procedures - see
the reference section.

[[remap:execution-sequencing]]
=== Execution sequencing

Executable G-code words are classified into <<gcode:modal-groups,modal groups>>,
which also defines their relative execution behavior.

If a  G-code block  contains several executable words on a line, these
words are executed in a predefined <<gcode:order-of-execution,order of execution>>,
not in the order they appear in block.

When you define a new executable code, the interpreter does not yet
know where your code fits into this scheme. For this
reason, you need to choose an appropriate modal group for your code to
execute in.

=== An minimal example remapped code

To give you an idea how the pieces fit together, let's explore a
fairly minimal but complete remapped code definition. We choose an
unallocated M-code and add the following option to the INI file:

[source,{ini}]
---------------------------------------------------------------------
[RS274NGC]
REMAP=M400  modalgroup=10 argspec=Pq ngc=myprocedure
---------------------------------------------------------------------

In a nutshell, this means:

- The `M400` code takes a required parameter `P` and an optional
  parameter `Q`. Other words in the current block are ignored with
  respect to the `M400` code. If the `P` word is not present, fail
  execution with an error.

- when an `M400` code is encountered, execute `myprocedure.ngc`  along
  the other <<gcode:modal-groups,modal group>> 10 M-codes as per
  <<gcode:order-of-execution,order of execution>>.

- the value of `P`, and `Q` are available in the procedure as local
  named parameters. The may be referred to  as `#<P>` and `#<Q>`. The
  procedure may test whether the `Q` word was present with the
  <<gcode:functions,`EXISTS`>> built in function.

The file `myprocedure.ngc` is expected to exists in the `[DISPLAY]NC_FILES` or
`[RS274NGC]SUBROUTINE_PATH` directory.

A detailed discussion of REMAP parameters is found in the reference
section below.

== Configuring Remapping

=== The REMAP statement

To remap a code, define it using the `REMAP` option in
`RS274NG` section of your INI file. Use one `REMAP` line per remapped code.

The syntax of the `REMAP` is:

`REMAP=`'<code>' '<options>'::
  where '<code>' may be one of `T`,`M6`,`M61`,`S`,`F` (existing codes) or any of the
  unallocated <<remap:unallocated-m-codes,M-codes>> or <<remap:unallocated-g-codes,G-codes>>.

It is an error to omit the '<code>' parameter.

The options of the REMAP statement are separated by whitespace. The options are
keyword-value pairs and currently are:

`modalgroup=`'<modal group>'::
  G-codes;; the only currently supported modal group is 1, which
  is also the default value if no group is given. Group 1 means
  'execute alongside other G-codes'.

  M-codes;; currently supported modal groups are:
    5,6,7,8,9,10. If no modalgroup is give, it defaults to 10
    ('execute after all other words in the block').

  T,S,F;; for these the modal group is fixed and any
    `modalgroup=` option is ignored.

`argspec=`'<argspec>'::
  See <<sub:argspec-parameter, description of the argspec
  parameter options>>. Optional.

`ngc=`'<ngc_basename>'::
  Basename of an O-word subroutine file name. Do not specify an
  .ngc extension. Searched for in the directories specified in
  the directory specified in `[DISPLAY]PROGRAM_PREFIX`, then in
  `[RS274NGC]SUBROUTINE_PATH`. Mutually exclusive with
  `python=`. It is an error to omit both `ngc=` and  `python=`.

`python=`'<Python function name>'::
  Instead of calling an ngc O-word procedure call a Python
  function. The function is expected to be defined in the
  `module_basename.oword`
  module. Mutually exclusive with `ngc=`.

`prolog=`'<Python function name>'::
  Before executing an ngc procedure, call this Python function.
  The function is expected to be defined in the
  `module_basename.remap`
  module. Optional.

`epilog=`'<Python function name>'::
  After executing an ngc procedure, call this Python function.
  The function is expected to be defined in the
  `module_basename.remap`
  module. Optional.

The `python`, `prolog` and `epilog` options require the Python
Interpreter plugin to be <<remap:embedded-python,configured>>, and
appropriate Python functions to be defined there so they can be
referred to with these options.

The syntax for defining a new code, and redefining an existing code is
identical.

=== Useful REMAP option combinations

Note that while many combinations of argspec options are possible, not
all of them make sense. The following combinations are useful idioms:

`argspec=`'<words>' `ngc=`'<procname>' `modalgroup=`'<group>'::
  The recommended way to call an NGC procedure with a standard argspec parameter
  conversion. Used if argspec is good enough. Note it's not good
  enough for remapping the Tx and M6/M61 tool change codes.

`prolog=`'<pythonprolog>' `ngc=`'<procname>' `epilog=`'<pythonepilog>' `modalgroup=`'<group>'::
  Call a Python prolog function to take any preliminary steps, then call the NGC
  procedure. When done, call the Python epilog function to do any
  cleanup or result extraction work which cannot be handled in G-code.
  The most flexible way of remapping a code to an NGC procedure,
  since almost all of the Interpreter internal variables, and some
  internal functions may be accessed from the prolog and epilog
  handlers. Also, a longer rope to hang yourselves.

`python=`'<pythonfunction>' `modalgroup=`'<group>'::
  Directly call to a Python function without any argument conversion.
  The most powerful way of remapping a code and going straight to
  Python. Use this if you don't need an NGC procedure, or NGC is
  just getting in your way.

`argspec=`'<words>' `python=`'<pythonfunction>' `modalgroup=`'<group>'::
  Convert the argspec words and pass them to a Python function as
  keyword argument dictionary. Use it when you're too lazy to
  investigate words passed on the block yourself.

Note that if all you want to achieve is to call some Python code from
G-code, there is the somewhat easier way of
<<remap:python-o-word-procs, calling Python functions like O-word procedures>>.

[[sub:argspec-parameter]]
=== The argspec parameter

The argument specification (keyword `argspec`) describes required and
optional words to be passed to an ngc procedure, as well as optional
preconditions for that code to execute.

An argspec consists of 0 or more  characters of the class
`[@A-KMNP-Za-kmnp-z^>]` . It can by empty (like `argspec=`).

An empty argspec, or no argspec argument at all implies the remapped
code does not receive  any parameters from the block. It will ignore
any extra parameters present.

Note that RS274NGC rules still apply - for instance you may use axis
words (eg X,Y,Z) only in the context of a G-code.

Axis words may also only be used if the axis is enabled.  If only XYZ
are enabled, ABCUVW will not be available to be used in argspec.

Words FST will have the normal functions but will be avaliable as 
variables in the remapped function. F will set feedrate, S will set
spindle RPM, T will trigger the tool prepare function.  Words FST 
should not be used if this behavior is not desired.

Words DEIJKPQR have no predefined function and are recommended for
use as argspec parameters.

`ABCDEFHIJKPQRSTUVWXYZ`::
  Defines a required word parameter: an uppercase letter specifies that
  the corresponding word *must*
  be present in the current block. The word`s value will be
  passed as a local named parameter with a corresponding name.
  If the `@` character is
  present in the argspec, it will be passed as positional
  parameter, see below.

`abcdefhijkpqrstuvwxyz`::
  Defines an optional word parameter: a lowercase letter specifies that
  the corresponding word *may* be present in the current block.
  If the word is present, the word's value will be
  passed as a local named parameter. If the `@` character is
  present in the argspec, it will be passed as positional
  parameter, see below.

`@`::
  The `@` (at-sign) tells argspec to pass words as positional
  parameters, in the order defined following the `@`
  option. Note that when using positional parameter passing,
  a procedure cannot tell whether a word was present or not, see
  example below.

TIP: this helps with packaging existing NGC procedures as remapped
codes. Existing procedures do expect positional parameters. With the
`@` option, you can avoid rewriting them to refer to local named
parameters.

`^`::
  The `^` (caret) character specifies that the current
  spindle speed must be greater than zero (spindle running),
  otherwise the code fails with an appropriate error message.

`>`::
  The `>` (greater-than) character specifies that the current
  feed must be greater than zero, otherwise the code fails with
  an appropriate error message.

`n`::
  The `n` (greater-than) character specifies to pass the current
  line number in the `n`local named parameter.

By default, parameters are passed  as local named parameter to an NGC
procedure. These local parameters appear as 'already set' when the
procedure starts executing, which is different from existing semantics
(local variables start out with value 0.0 and need to be explicitly
assigned a value).

Optional word parameters may be tested for presence by the `EXISTS(#<word>)` idiom.

.Example for named parameter passing to NGC procedures

Assume the code is defined as

`REMAP=M400  modalgroup=10 argspec=Pq ngc=m400`

and `m400.ngc` looks as follows:

[source,{ngc}]
----------------------------------------------------------------------------------
o<m400> sub
(P is required since it's uppercase in the argspec)
(debug, P word=#<P>)
(the q argspec is optional since its lowercase in the argspec. Use as follows:)
o100 if [EXISTS[#<q>]]
    (debug, Q word set: #<q>)
o100 endif
o<m400> endsub
M2
----------------------------------------------------------------------------------

- executing `M400` will fail with the message `user-defined M400: missing: P`
- executing `M400 P123` will display `P word=123.000000`
- executing `M400 P123 Q456` will display `P word=123.000000` and `Q word set: 456.000000`

.Example for positional parameter passing to NGC procedures

Assume the code is defined as

`REMAP=M410  modalgroup=10 argspec=@PQr ngc=m410`

and `m410.ngc` looks as follows:

[source,{ngc}]
----------------------------------------------------------------------------------
o<m410> sub
(debug, [1]=#1 [2]=#2 [3]=#3)
o<m410> endsub
M2
----------------------------------------------------------------------------------

- executing `M410 P10` will display `m410.ngc: [1]=10.000000 [2]=0.000000`
- executing `M410 P10 Q20` will display `m410.ngc: [1]=10.000000 [2]=20.000000`

NOTE: you lose the capability to distinguish more than one optional
parameter word, and you cannot tell whether an optional parameter was
present but had the value 0, or was not present at all.

.Simple example for named parameter passing to a Python function

It's possible to define new codes 'without' any NGC procedure. Here's
a simple first example, a more complex one can be found in the next
section.

Assume the code is defined as

`REMAP=G88.6 modalgroup=1  argspec=XYZp  python=g886`

This instructs the interpreter to execute the Python function `g886`
in the `module_basename.remap` module
which might look like so:

[source,python]
---------------------------------------------------------------------
from interpreter import INTERP_OK
from emccanon import MESSAGE

def g886(self, **words):
    for key in words:
        MESSAGE("word '%s' = %f" % (key, words[key]))
    if words.has_key('p'):
        MESSAGE("the P word was present")
    MESSAGE("comment on this line: '%s'" % (self.blocks[self.remap_level].comment))
    return INTERP_OK
---------------------------------------------------------------------

Try this with out with:
  g88.6 x1 y2 z3
  g88.6 x1 y2 z3 p33 (a comment here)

You'll notice the gradual introduction of the embedded Python
environment - see <<remap:programming-embedded-python,here>> for details.  Note that
with Python remapping functions, it make no sense to have Python
prolog or epilog functions since it's executing a Python function in
the first place.

.Advanced example: Remapped codes in pure Python

The `interpreter` and `emccanon` modules expose most of the Interpreter
and some Canon internals, so many things which  so far required coding in
'C/C\+\+' can be now be done in Python.

The following example is based on the `nc_files/involute.py` script -
but canned as a G-code with some parameter extraction and checking. It
also demonstrates calling the interpreter recursively (see `self.execute()`).

Assuming a definition like so (NB: this does not use argspec):

`REMAP=G88.1 modalgroup=1  py=involute`

The `involute` function in `python/remap.py` listed below does all
word extraction from the current block directly. Note that interpreter
errors can be translated to Python exceptions. Remember this is
'readahead time' - execution time errors cannot be trapped this way.

[source,python]
---------------------------------------------------------------------
import sys
import traceback
from math import sin,cos

from interpreter import *
from emccanon import MESSAGE
from util import lineno, call_pydevd
# raises InterpreterException if execute() or read() fails
throw_exceptions = 1

def involute(self, **words):
    """ remap function with raw access to Interpreter internals """

    if self.debugmask & 0x20000000: call_pydevd() # USER2 debug flag

    if equal(self.feed_rate,0.0):
        return "feedrate > 0 required"

    if equal(self.speed,0.0):
        return "spindle speed > 0 required"

    plunge = 0.1 # if Z word was given, plunge - with reduced feed

    # inspect controlling block for relevant words
    c = self.blocks[self.remap_level]
    x0 = c.x_number if c.x_flag else 0
    y0 = c.y_number if c.y_flag else 0
    a  = c.p_number if c.p_flag else 10
    old_z = self.current_z

    if self.debugmask & 0x10000000:
        print("x0=%f y0=%f a=%f old_z=%f" % (x0,y0,a,old_z))

    try:
        #self.execute("G3456")  # would raise InterpreterException
        self.execute("G21",lineno())
        self.execute("G64 P0.001",lineno())
        self.execute("G0 X%f Y%f" % (x0,y0),lineno())

        if c.z_flag:
            feed = self.feed_rate
            self.execute("F%f G1 Z%f" % (feed * plunge, c.z_number),lineno())
            self.execute("F%f" % (feed),lineno())

        for i in range(100):
            t = i/10.
            x = x0 + a * (cos(t) + t * sin(t))
            y = y0 + a * (sin(t) - t * cos(t))
            self.execute("G1 X%f Y%f" % (x,y),lineno())

        if c.z_flag: # retract to starting height
            self.execute("G0 Z%f" % (old_z),lineno())

    except InterpreterException,e:
        msg = "%d: '%s' - %s" % (e.line_number,e.line_text, e.error_message)
  return msg

    return INTERP_OK
---------------------------------------------------------------------

The examples described so far can be found in
'configs/sim/axis/remap/getting-started' with complete working
configurations.

[[remap:upgrading-an-existing]]
== Upgrading an existing configuration for remapping

The minimal prerequisites for using `REMAP` statements are as follows:

- the Python plug in must be activated by specifying a
  `[PYTHON]TOPLEVEL=<path-to-toplevel-script>` in the INI file.
- the toplevel script needs to import the `remap` module, which can be
  initially empty, but the import needs to be in place.
- The Python interpreter needs to find the remap.py module above, so
  the path to the directory where your Python modules live needs to be
  added with  `[PYTHON]PATH_APPEND=<path-to-your-local-Python-directory>`
- Recommended: import the `stdglue` handlers in the `remap` module. In
  this case Python also needs to find `stdglue.py` - we just copy it
  from the distribution so you can make local changes as
  needed. Depending on your installation the path to `stdglue.py` might
  vary.

Assuming your configuration lives under `/home/user/xxx` and the INI
file is `/home/user/xxx/xxx.ini`, execute the following commands.

[source,sh]
---------------------------------------------------------------------
$ cd /home/user/xxx
$ mkdir python
$ cd python
$ cp /usr/share/linuxcnc/ncfiles/remap_lib/python-stdglue/stdglue.py .
$ echo 'from stdglue import *' >remap.py
$ echo 'import remap' >toplevel.py
---------------------------------------------------------------------

Now edit `/home/user/xxx/xxx.ini` and add the following:

[source,{ini}]
---------------------------------------------------------------------
[PYTHON]
TOPLEVEL=/home/user/xxx/python/toplevel.py
PATH_APPEND=/home/user/xxx/python
---------------------------------------------------------------------

Now verify that LinuxCNC comes up with no error messages - from a
terminal window execute:

[source,sh]
---------------------------------------------------------------------
$ cd /home/user/xxx
$ linuxcnc xxx.ini
---------------------------------------------------------------------

== Remapping tool change-related codes: T, M6, M61

=== Overview

If you are unfamiliar with LinuxCNC internals, first read the
<<remap:how-tool-change-currently-works,How tool change currently works>>
section (dire but necessary).

Note than when remapping an existing code, we completely disable
<<remap:interpreter-action-on-m6,this codes' built in functionality>>
of the interpreter.

So our remapped code will need to do a bit more
than just generating some commands to move the machine as we like - it
will also need to replicate those steps from this sequence which are
needed to keep the interpreter and `task` happy.

However, this does *not* affect the processing of
tool change-related commands in `task` and `iocontrol`. This means when we
execute <<remap:send-tool-load-msg,step 6b>> this will still cause
<<remap:iocontrol-action-on-load,iocontrol to do its thing>>.

Decisions, decisions:

- Do we want to use an O-word procedure or do it all in Python code?
- Is the `iocontrol` HAL sequence (tool-prepare/tool-prepared and
  tool-change/tool-changed pins) good enough or do we need a different kind
  of HAL interaction for our tool changer (for example: more HAL pins
  involved with a  different interaction sequence)?

Depending on the answer, we have four different scenarios:

- When using an O-word procedure, we need prolog and epilog functions
- if using all Python code and no O-word procedure, a Python function
  is enough
- when using the `iocontrol` pins, our O-word procedure or Python code
  will contain mostly moves
- when we need a more complex interaction than offered by `iocontrol`,
  we need to completely define our own interaction, using `motion.digital*` and
  `motion.analog*` pins, and essentially ignore the `iocontrol` pins by
  looping them.

NOTE: If you hate O-word procedures and love Python, you're free to do it
all in Python, in which case you would just have a `python=<function>`
spec in the REMAP statement. But assuming most folks would be interested in
using O-word procedures because they are more familiar with that, we'll do
that as the  first example.

So the overall approach for our first example will be:

. we'd like to do as much as possible with G-code in an O-word
  procedure for flexibility. That includes all HAL interaction which
  would normally be handled by `iocontrol` - because we rather would want to do
  clever things with moves, probes, HAL pin I/O and so forth.

. we'll try to minimize Python code to the extent needed  to keep the interpreter happy,
  and cause `task` to actually do anything. That will go into the
  `prolog` and `epilog` Python functions.

=== Understanding the role of `iocontrol` with remapped tool change codes

`iocontrol` provides two HAL interaction sequences we might or might not
use:

- when the NML message queued by a SELECT_TOOL() canon command is
  executed, this triggers the "raise tool-prepare and wait for
  tool-prepared to become high" HAL sequence in `iocontrol`, besides
  setting the XXXX pins
- when the NML message queued by the CHANGE_TOOL() canon command is
  executed, this triggers the  "raise tool-change and wait for
  tool-changed to become high" HAL sequence in `iocontrol`, besides
  setting the XXXX pins

What you need to decide is whether the existing `iocontrol` HAL sequences
are sufficient to drive your changer. Maybe you need a different
interaction sequence - for instance more HAL  pins, or maybe a more
complex interaction. Depending on the answer, we might continue to use the existing
`iocontrol` HAL sequences, or define our own ones.

For the sake of documentation, we'll disable these `iocontrol`
sequences, and roll our own - the result will look and feel like the
existing interaction, but now we have complete control over them
because they are executed in our own O-word procedure.

So what we'll do is use some `motion.digital-*` and `motion.analog-*`
pins, and the associated `M62` .. `M68` commands to do our own HAL
interaction in our O-word procedure, and those will effectively
replace the `iocontrol` 'tool-prepare/tool-prepared' and
'tool-change/tool-changed' sequences. So we'll define our pins
replacing existing `iocontrol` pins functionally, and go ahead and make
the `iocontrol` interactions a loop. We'll use the following
correspondence in our example:

`iocontrol` pin correspondence in the examples

[width="60%",cols="2"]
[frame="topbot",grid="none"]
[options="header"]
|===
|iocontrol.0 pin|motion pin
|tool-prepare|digital-out-00
|tool-prepared|digital-in-00
|tool-change|digital-out-01
|tool-changed|digital-in-01
|tool-prep-number|analog-out-00
|tool-prep-pocket|analog-out-01
|tool-number|analog-out-02
|===

Let us assume you want to redefine the M6 command, and replace it by
an O-word procedure, but other than that things 'should continue to
work'.

So what our O-word procedure would do is to replace the steps
<<remap:interpreter-action-on-m6,outlined here>>. Looking through
these steps you'll find that NGC code can be used for most of them,
but not all. So the stuff NGC can't handle will be done in Python prolog
and epilog functions.

=== Specifying the M6 replacement

To convey the idea, we just replace the built in M6 semantics with our
own. Once that works, you may go ahead and place any actions you see
fit into the O-word procedure.

Going through the <<remap:interpreter-action-on-m6,steps>>, we find:

. check for T command already executed - *execute in Python prolog*
. check for cutter compensation being active - *execute in Python prolog*
. stop the spindle if needed - *can be done in NGC*
. quill up - *can be done in NGC*
. if TOOL_CHANGE_AT_G30 was set:
.. move the A, B and C indexers if applicable - *can be done in NGC*
.. generate rapid move to the G30 position - *can be done in NGC*
. send a CHANGE_TOOL Canon command to `task`  - *execute in Python epilog*
. set the numberer parameters 5400-5413 according to the new tool - *execute in Python epilog*
. signal to `task` to stop calling the interpreter for readahead until
  tool change complete - *execute in Python epilog*

So we need a prolog, and an epilog. Lets assume our INI file incantation of the M6 remap looks as follows:

----
REMAP=M6   modalgroup=6  prolog=change_prolog ngc=change epilog=change_epilog
----

So the prolog covering steps 1 and 2 would look like so - we decide to
pass a few variables to the remap procedure which can be inspected and
changed there, or used in a message. Those are: `tool_in_spindle`,
`selected_tool` (tool numbers) and their respective tooldata indices
`current_pocket` and `selected_pocket`:

[NOTE]
The legacy names *selected_pocket* and *current_pocket* actually reference
a sequential tooldata index for tool items loaded from a tool
table ([EMCIO]TOOL_TABLE) or via a tooldata database ([EMCIO]DB_PROGRAM)

[source,python]
---------------------------------------------------------------------
def change_prolog(self, **words):
    try:
        if self.selected_pocket < 0:
            return "M6: no tool prepared"

        if self.cutter_comp_side:
            return "Cannot change tools with cutter radius compensation on"

        self.params["tool_in_spindle"] = self.current_tool
        self.params["selected_tool"] = self.selected_tool
        self.params["current_pocket"] = self.current_pocket
        self.params["selected_pocket"] = self.selected_pocket
        return INTERP_OK
    except Exception, e:
        return "M6/change_prolog: %s" % (e)
---------------------------------------------------------------------

You will find that most prolog functions look very similar: first test
that all preconditions for executing the code hold, then prepare the
environment - inject variables and/or do any preparatory processing
steps which cannot easily be done in NGC code; then hand off to the
NGC procedure by returning INTERP_OK.

Our first iteration of the O-word procedure is unexciting - just
verify we got parameters right, and signal success by returning a
positive value; steps 3-5 would eventually be covered here (see
<<gcode:ini-hal-params,here>> for the variables referring
to INI file settings):

[source,{ngc}]
---------------------------------------------------------------------
O<change> sub
(debug, change: current_tool=#<current_tool>)
(debug, change: selected_pocket=#<selected_pocket>)
;
; insert any G-code which you see fit here, eg:
; G0  #<_ini[setup]tc_x>  #<_ini[setup]tc_y>  #<_ini[setup]tc_z>
;
O<change> endsub [1]
m2
---------------------------------------------------------------------

Assuming success of `change.ngc`, we need to mop up steps 6-8:

[source,python]
---------------------------------------------------------------------
def change_epilog(self, **words):
    try:
        if self.return_value > 0.0:
            # commit change
            self.selected_pocket =  int(self.params["selected_pocket"])
            emccanon.CHANGE_TOOL(self.selected_pocket)
            # cause a sync()
            self.tool_change_flag = True
            self.set_tool_parameters()
            return INTERP_OK
        else:
            return "M6 aborted (return code %.1f)" % (self.return_value)

    except Exception, e:
        return "M6/change_epilog: %s" % (e)
---------------------------------------------------------------------

This replacement M6 is compatible with the built in code,
except steps 3-5 need to be filled in with your NGC code.

Again, most epilogs have a common scheme: first, determine whether
things went right in the remap procedure, then do any commit and cleanup
actions which can't be done in NGC code.

=== Configuring `iocontrol` with a remapped M6

Note that the sequence of operations has changed: we do everything
required in the O-word procedure - including any HAL pin
setting/reading to get a changer going, and to acknowledge a tool
change - likely with `motion.digital-*` and `motion-analog-*` IO
pins. When we finally execute the `CHANGE_TOOL()` command, all
movements and HAL interactions are already completed.

Normally only now `iocontrol` would do its thing as outlined
<<remap:iocontrol-action-on-load,here>>. However, we don't need the
HAL pin wiggling anymore - all `iocontrol` is left to do is to accept
we're done with prepare and change.

This means that the corresponding `iocontrol` pins have no function any
more. Therefore, we configure `iocontrol` to immediately acknowledge a
change by configuring like so:

[source,{hal}]
---------------------------------------------------------------------
# loop change signals when remapping M6
net tool-change-loop iocontrol.0.tool-change iocontrol.0.tool-changed
---------------------------------------------------------------------
If you for some reason want to remap `Tx` (prepare), the corresponding
`iocontrol` pins need to be looped as well.

=== Writing the change and prepare O-word procedures

The standard prologs and epilogs found in
`ncfiles/remap_lib/python-stdglue/stdglue.py` pass a few 'exposed
parameters' to the remap procedure.

An 'exposed parameter' is a named local variable visible in a remap
procedure which corresponds to interpreter-internal variable which
is relevant for the current remap. Exposed parameters
are set up in the respective prolog, and inspected in the epilog. They
can be changed in the remap procedure and the change will be picked up
in the epilog. The exposed parameters for remappable built in codes are:

- `T` (prepare_prolog): `#<tool>` , `#<pocket>`
- `M6` (change_prolog): `#<tool_in_spindle>`, `#<selected_tool>`, `#<current_pocket>`, `#<selected_pocket>`
- `M61` (settool_prolog): `#<tool>` , `#<pocket>`
- `S` (setspeed_prolog):  `#<speed>`
- `F` (setfeed_prolog):  `#<feed>`

If you have specific needs for extra parameters to be made visible,
that can simply be added to the prolog - practically all of the
interpreter internals are visible to Python.

=== Making minimal changes to the built in codes, including `M6`

Remember that normally remapping a code completely disables all internal
processing for that code.

However, in some situations it might be sufficient to add a few codes around
the existing `M6` built in implementation, like a tool length probe,
but other than that retain the behavior of the built in `M6`.

Since this might be a common scenario, the built in behavior of
remapped codes has been made available within the remap
procedure. The interpreter detects that you are referring to a
remapped code within the procedure which is supposed to redefine its
behavior. In this case, the built in behavior is used - this
currently is enabled for the set: `M6`, `M61`,`T`, `S`, `F`. Note
that otherwise referring to a code within its own remap procedure
would be a error - a `remapping recursion`.

Slightly twisting a built in would look like so (in the case of `M6`):

----
REMAP=M6   modalgroup=6  ngc=mychange
----

[source,{ngc}]
---------------------------------------------------------------------
o<mychange> sub
M6 (use built in M6 behavior)
(.. move to tool length switch, probe and set tool length..)
o<mychange> endsub
m2
---------------------------------------------------------------------

CAUTION: when redefining a built in code, *do not specify any leading
zeroes in G- or M-codes* - for example, say `REMAP=M1 ..`, not
`REMAP=M01 ...`.

See the `configs/sim/axis/remap/extend-builtins` directory for a complete
configuration which is the recommended starting point for own work
when extending built in codes.

=== Specifying the T (prepare) replacement

If you're confident with the  <<remap:interpreter-action-on-t,default
implementation>>, you wouldn't need to do this. But remapping is also a
way to work around deficiencies in the current implementation, for
instance to not block until the "tool-prepared" pin is set.

What you could do, for instance, is:
- in a remapped T, just set the equivalent of the "tool-prepare" pin,
  but *not* wait for "tool-prepared" here
- in the corresponding remapped M6, wait for the "tool-prepared" at
  the very beginning of the O-word procedure.

Again, the `iocontrol` tool-prepare/tool-prepared pins would be unused
and replaced by `motion.*` pins, so those would pins must be looped:

[source,{hal}]
---------------------------------------------------------------------
# loop prepare signals when remapping T
net tool-prep-loop iocontrol.0.tool-prepare iocontrol.0.tool-prepared
---------------------------------------------------------------------

So, here's the setup for a remapped T:

----
REMAP=T  prolog=prepare_prolog epilog=prepare_epilog ngc=prepare
----

[source,python]
---------------------------------------------------------------------
def prepare_prolog(self,**words):
    try:
        cblock = self.blocks[self.remap_level]
        if not cblock.t_flag:
            return "T requires a tool number"

        tool  = cblock.t_number
        if tool:
            (status, pocket) = self.find_tool_pocket(tool)
            if status != INTERP_OK:
                return "T%d: pocket not found" % (tool)
        else:
            pocket = -1 # this is a T0 - tool unload

        # these variables will be visible in the ngc O-word sub
        # as #<tool> and #<pocket> local variables, and can be
        # modified there - the epilog will retrieve the changed
        # values
        self.params["tool"] = tool
        self.params["pocket"] = pocket

        return INTERP_OK
    except Exception, e:
        return "T%d/prepare_prolog: %s" % (int(words['t']), e)

---------------------------------------------------------------------

The minimal ngc prepare procedure again looks like so:

[source,{ngc}]
---------------------------------------------------------------------
o<prepare> sub
; returning a positive value to commit:
o<prepare> endsub [1]
m2
---------------------------------------------------------------------

And the epilog:

[source,python]
---------------------------------------------------------------------
def prepare_epilog(self, **words):
    try:
        if self.return_value > 0:
            self.selected_tool = int(self.params["tool"])
            self.selected_pocket = int(self.params["pocket"])
            emccanon.SELECT_TOOL(self.selected_tool)
            return INTERP_OK
        else:
            return "T%d: aborted (return code %.1f)" % (int(self.params["tool"]),self.return_value)

    except Exception, e:
        return "T%d/prepare_epilog: %s" % (tool,e)

---------------------------------------------------------------------

prepare_prolog and prepare_epilog are part of the 'standard glue'
provided by 'nc_files/remap_lib/python-stdglue/stdglue.py'.
This module is intended to cover most
standard remapping situations in a common way.

=== Error handling: dealing with abort

The built in tool change procedure has some precautions for dealing
with a program abort (e.g. hitting Escape in Axis during a
change). Your remapped function has none of this, therefore some
explicit cleanup might be needed if a remapped code is aborted. In
particular, a remap procedure might establish modal settings which are
undesirable to have active after an abort. For instance, if your
remap procedure has motion codes (G0,G1,G38..) and the remap is
aborted, then the last modal code will remain active. However, you
very likely want to have any modal motion canceled when the remap is
aborted.

The way to do this is by using the `[RS274NGC]ON_ABORT_COMMAND`
feature. This INI option specifies a O-word procedure call which is
executed if `task` for some reason aborts program execution.
`on_abort` receives a single parameter indicating 
the cause for calling the abort procedure,
which might be used for conditional cleanup.

The reasons are defined in nml_intf/emc.hh

[source,{ini}]
---------------------------------------------------------------------
	EMC_ABORT_TASK_EXEC_ERROR = 1,
	EMC_ABORT_AUX_ESTOP = 2,
	EMC_ABORT_MOTION_OR_IO_RCS_ERROR = 3,
	EMC_ABORT_TASK_STATE_OFF = 4,
	EMC_ABORT_TASK_STATE_ESTOP_RESET = 5,
	EMC_ABORT_TASK_STATE_ESTOP = 6,
	EMC_ABORT_TASK_STATE_NOT_ON = 7,
	EMC_ABORT_TASK_ABORT = 8,
	EMC_ABORT_INTERPRETER_ERROR = 9,	// interpreter failed during readahead
	EMC_ABORT_INTERPRETER_ERROR_MDI = 10,	// interpreter failed during MDI execution
	EMC_ABORT_USER = 100  // user-defined abort codes start here
---------------------------------------------------------------------


[source,{ini}]
---------------------------------------------------------------------
[RS274NGC]
ON_ABORT_COMMAND=O <on_abort> call
---------------------------------------------------------------------

The suggested on_abort procedure would look like so (adapt to
your needs):

[source,{ngc}]
---------------------------------------------------------------------
o<on_abort> sub

G54 (origin offsets are set to the default)
G17 (select XY plane)
G90 (absolute)
G94 (feed mode: units/minute)
M48 (set feed and speed overrides)
G40 (cutter compensation off)
M5  (spindle off)
G80 (cancel modal motion)
M9  (mist and coolant off)

o100 if [#1 eq 5]
    (machine on)
o100 elseif [#1 eq 6]
    (machine off)
o100 elseif [#1 eq 7]
    (estopped)
o100 elseif [#1 eq 8]
    (msg, abort pressed)
o100 else
    (DEBUG, error parameter is [#1])
o100 endif

o<on_abort> endsub
m2
---------------------------------------------------------------------

CAUTION: Never use an `M2` in a O-word subroutine, including this
one. It will cause hard-to-find errors. For instance, using an `M2` in
a subroutine will not end the subroutine properly and will leave the
subroutine NGC file open, not your main program.

Make sure `on_abort.ngc` is along the interpreter search path
(recommended location: `SUBROUTINE_PATH` so as not to clutter your
`NC_FILES` directory with internal procedures).

Statements in that procedure typically would assure that post-abort
any state has been cleaned up, like HAL pins properly reset. For an
example, see `configs/sim/axis/remap/rack-toolchange`.

Note that terminating a remapped code by returning INTERP_ERROR from
the epilog (see previous section) will also cause the `on_abort` procedure
to be called.

=== Error handling: failing a remapped code NGC procedure

If you determine in your handler procedure that some error condition
occurred, do not use `M2` to end your handler - see above:

If displaying an operator error message and stopping the current program is
good enough, use the `(abort, <message>)` feature to terminate the handler with an
error message. Note that you can substitute numbered, named, INI and
HAL parameters in the text like
in this example (see also `tests/interp/abort-hot-comment/test.ngc`):

[source,{ngc}]
---------------------------------------------------------------------
o100 if [..] (some error condition)
     (abort, Bad Things! p42=#42 q=#<q> ini=#<_ini[a]x> pin=#<_hal[component.pin])
o100 endif
---------------------------------------------------------------------

NOTE: INI and HAL variable expansion is optional and can be disabled in
the <<sub:ini:sec:rs274ngc,INI file>>

If more fine grained recovery action is needed, use the idiom
laid out in the previous example:

- define an epilog function, even if it's just to signal an error
  condition
- pass a negative value from the handler to signal
  the error
- inspect the return value in the epilog function.
- take any recovery action needed
- return the error message string from the handler, which will set the
  interpreter error message and abort the program (pretty much like
  `abort, message=`)

This error message will be displayed in the UI, and returning
INTERP_ERROR will cause this error handled like any other runtime error.

Note that both `(abort, msg)` and returning INTERP_ERROR from an
epilog will cause any ON_ABORT handler to be called as well if defined
(see previous section).

== Remapping other existing codes: S, M0, M1, M60

=== Automatic gear selection be remapping  S (set spindle speed)

A potential use for a remapped S code would be 'automatic gear
selection' depending on speed. In the remap procedure one would test
for the desired speed attainable given the current gear setting, and
change gears appropriately if not.

=== Adjusting the behavior of M0, M1, M60

A use case for remapping M0/M1 would be to customize the behavior of
the existing code. For instance, it could be desirable to turn off the
spindle, mist and flood during an M0 or M1 program pause, and turn
these settings back on when the program is resumed.

For a complete example doing just that, see
'configs/sim/axis/remap/extend-builtins/', which adapts M1 as laid out above.

== Creating new G-code cycles

A G-code cycle as used here is meant to behave as follows:

* On first invocation, the associated words are collected and the
  G-code cycle is executed.
* If subsequent lines just continue parameter words applicable to this
  code, but no new G-code, the previous G-code is re-executed with the
  parameters changed accordingly.

An example: Assume you have `G84.3` defined as remapped G-code cycle
with the following INI segment (see <<remap:cycle-stdglue,here>> for
a detailed description of +cycle_prolog+ and +cycle_epilog+):

[source,{ini}]
---------------------------------------------------------------------
[RS274NGC]
# A cycle with an O-word procedure: G84.3 <X- Y- Z- Q- P->
REMAP=G84.3 argspec=xyzabcuvwpr prolog=cycle_prolog ngc=g843 epilog=cycle_epilog modalgroup=1
---------------------------------------------------------------------

Executing the following lines:

[source,{ngc}]
---------------------------------------------------------------------
g17
(1)   g84.3 x1 y2 z3  r1
(2)   x3 y4 p2
(3)   x6 y7 z5
(4)   G80
---------------------------------------------------------------------

causes the following (note 'R' is sticky, and 'Z' is sticky since the plane is 'XY'):

. `g843.ngc` is called with words x=1, y=2, z=3, r=1
. `g843.ngc` is called with words x=3, y=4, z=3, p=2, r=1
. `g843.ngc` is called with words x=6, y=7, z=3, r=1
.  The `G84.3` cycle is canceled.

Besides creating new cycles, this provides an easy method for
repackaging existing G-codes which do not behave as cycles. For
instance, the `G33.1` Rigid Tapping code does not behave as a
cycle. With such a wrapper, a new code can be easily created which
uses `G33.1` but behaves as a cycle.

See 'configs/sim/axis/remap/cycle' for a complete example of this
feature. It contains two cycles, one with an NGC procedure like above,
and a cycle example using just Python.

[[remap:embedded-python]]
== Configuring  Embedded Python

The Python plugin serves both the interpreter, and `task` if so
configured, and hence has its own section `PYTHON` in the INI file.

=== Python plugin : INI file configuration

`[PYTHON]`::
  `TOPLEVEL =` <filename>;;
    filename of the initial Python script to execute on
    startup. This script is responsible for setting up the package
    name structure, see below.
  `PATH_PREPEND =` <directory>;;
    prepend this directory to `PYTHON_PATH`. A repeating
    group.
  `PATH_APPEND =`<directory>;;
    append this directory to `PYTHON_PATH`. A repeating
    group.
  `LOG_LEVEL =` <integer>;;
    log level of plugin-related actions. Increase this if you
    suspect problems. Can be very verbose.
  `RELOAD_ON_CHANGE =` [0|1];;
    reload the 'TOPLEVEL' script if the file was changed. Handy
    for debugging but currently incurs some runtime overhead. Turn
    this off for production configurations.
  `PYTHON_TASK =` [0|1];;
    Start the Python task plug in. Experimental. See xxx.

[[remap:executing-python-statements]]
=== Executing Python statements from the interpreter

For ad-hoc execution of commands the Python 'hot comment' has been
added. Python output by default goes to stdout, so you need to start
LinuxCNC from a terminal window to see results. Example (eg. in the
MDI window):

----
;py,print(2*3)
----

Note that the interpreter instance is available here as `self`, so you
could also run:

----
;py,print(self.tool_table[0].toolno)
----

The `emcStatus` structure is accessible, too:

----
;py,from emctask import *
;py,print(emcstat.io.aux.estop)
----

[[remap:programming-embedded-python]]
== Programming Embedded Python in the RS274NGC Interpreter

=== The Python plugin namespace

The namespace is expected to be laid out as follows:

`oword`::
  Any callables in this module are candidates for Python O-word
  procedures. Note that the Python `oword` module is checked
  *before* testing for a NGC procedure with the same name - in
  effect names in `oword` will hide NGC files of the same
  basename.

`remap`::
  Python callables referenced in an argspec `prolog`,`epilog` or
  `python` option are expected to be found here.

`namedparams`::
  Python funtcions int this module extend or redefine the namespace of
  predefined named parameters, see
  <<remap:adding-predefined-named-parameters,adding predefined parameters>>.

`task`::
  Task-related callables are expected here.

=== The Interpreter as seen from Python

The interpreter is an existing C++ class ('Interp') defined in
'src/emc/rs274ngc'. Conceptually all `oword.<function>` and
`remap.<function>` Python calls are methods of this Interp class,
although there is no explicit Python definition of this class (it's a
'Boost.Python' wrapper instance) and hence receive the as the first
parameter `self` which can be used to access internals.

=== The Interpreter `__init__` and `__delete__` functions

If the `TOPLEVEL`  module defines a function `__init__`, it will be
called once the interpreter is fully configured (INI file read, and
state synchronized with the world model).

If the `TOPLEVEL`  module defines a function `__delete__`, it will be
called once before the interpreter is shutdown and after the persistent
parameters have been saved to the `PARAMETER_FILE`.

Note_ at this time, the `__delete__` handler does not work for
interpreter instances created by importing the `gcode` module. If you
need an equivalent functionality there (which is quite unlikely),
please consider the Python `atexit` module.

[source,python]
---------------------------------------------------------------------
# this would be defined in the TOPLEVEL module

def __init__(self):
    # add any one-time initialization here
    if self.task:
  # this is the milltask instance of interp
  pass
    else:
  # this is a non-milltask instance of interp
        pass

def __delete__(self):
    # add any cleanup/state saving actions here
    if self.task: # as above
  pass
    else:
        pass
---------------------------------------------------------------------

This function may be used to initialize any Python-side attributes
which might be needed later, for instance in remap or oword
functions, and save or restore state beyond what `PARAMETER_FILE` provides.

If there are setup or cleanup actions which are to happen only in the
milltask Interpreter instance (as opposed to the interpreter instance
which sits in the `gcode` Python module and serves preview/progress
display purposes but nothing else), this can be tested for by
<<remap:axis-preview-and-remapped-code-execution,evaluating 'self.task'>>.

An example use of `__init__` and `__delete__` can be found in
'configs/sim/axis/remap/cycle/python/toplevel.py' initialising attributes
needed to handle cycles in 'ncfiles/remap_lib/python-stdglue/stdglue.py'
(and imported into 'configs/sim/axis/remap/cycle/python/remap.py').

=== Calling conventions: NGC to Python

Python code is called from NGC in the following situations:

- during normal program execution:
  * when an O-word call like `O<proc> call` is executed and the name
    `oword.proc` is defined and callable
  * when a comment like `;py,<Python statement>` is executed
- during execution of a remapped code: any `prolog=`, `python=` and
  `epilog=` handlers.

[[remap:python-o-word-procs]]
.Calling O-word Python subroutines

Arguments:

`self`::
  the interpreter instance

`*args`::
  the list of actual positional parameters. Since the number
  of actual parameters may vary, it is best to use this style of declaration:

[source,python]
---------------------------------------------------------------------
# this would be defined in the oword module
def mysub(self, *args):
    print("number of parameters passed:", len(args))
    for a in args:
  print(a)
---------------------------------------------------------------------

.Return values of O-word Python subroutines
Just as NGC procedures may return values, so do O-word Python
subroutines. They are expected to either:

- return no value (no `return` statement or the value `None`)
- a float or int value
- a string, this  means 'this is an error message, abort the program'. Works like `(abort, msg)`.

Any other return value type will raise a Python exception.

In a calling NGC environment, the following predefined named parameters
are available:

`#<_value>`::
  value returned by the last procedure called. Initialized
  to 0.0 on startup. Exposed in Interp as `self.return_value` (float).

`#<_value_returned>`::
  indicates the last procedure called did `return`or
  `endsub` with an explicit value. 1.0 if true. Set to 0.0 on each `call`. Exposed
  in Interp was  `self.value_returned` (int).

See also `tests/interp/value-returned` for an example.

.Calling conventions for 'prolog=' and 'epilog=' subroutines

Arguments are:

`self`::
  the interpreter instance

`words`::
  keyword parameter dictionary. If an argspec was present, words
  are collected from the current block accordingly and passed in
  the dictionary for  convenience (the words could as well be
  retrieved directly from the  calling block, but this requires
  more knowledge of interpreter internals). If no argspec was
  passed, or only optional values were specified and none of
  these was present in the calling block, this dict is
  empty. Word names are converted to lowercase.

Example call:

[source,python]
---------------------------------------------------------------------
def minimal_prolog(self, **words): # in remap module
    print(len(words)," words passed")
    for w in words:
        print("%s: %s" % (w, words[w]))
    if words['p'] < 78: # NB: could raise an exception if p were optional
       return "failing miserably"
    return INTERP_OK
---------------------------------------------------------------------

Return values:

`INTERP_OK`::
  return this on success. You need to import this from `interpreter`.

`"a message text"`::
  returning a string from a handler means 'this is an error message,
  abort the program'. Works like `(abort, msg)`.

.Calling conventions for 'python=' subroutines

Arguments are:

`self`::
  the interpreter instance

`words`::
  keyword parameter dictionary. the same  kwargs dictionary as
  prologs and epilogs (see above).

The minimum `python=` function example:

[source,python]
---------------------------------------------------------------------
def useless(self,  **words): # in remap module
    return INTERP_OK
---------------------------------------------------------------------

Return values:

`INTERP_OK`::
  return this on success

`"a message text"`::
  returning a string from a handler means 'this is an error message,
  abort the program'. Works like `(abort, msg)`.

If the handler needs to execute a 'queuebuster
operation' (tool change, probe, HAL pin reading) it is supposed
to suspend execution with the following statement:

`yield INTERP_EXECUTE_FINISH`::
  This signals `task` to stop read ahead, execute all
  queued operations, execute the 'queue-buster' operation,
  synchronize interpreter state with machine state, and then signal
  the interpreter to continue. At this point the function is
  resumed at the statement following the `yield ..` statement.

.Dealing with queue-buster: Probe, Tool change and waiting for a HAL pin

Queue busters interrupt a procedure at the point where such an
operation is called, hence the procedure needs to be restarted
after the interpreter synch(). When this happens the procedure needs to
know if it is restarted, and where to continue. The Python generator
method is used to deal with procedure restart.

This demonstrates call continuation with a single point-of-restart:

[source,python]
---------------------------------------------------------------------
def read_pin(self,*args):
    # wait 5secs for digital-input 00 to go high
    emccanon.WAIT(0,1,2,5.0)
    # cede control after executing the queue buster:
    yield INTERP_EXECUTE_FINISH
    # post-sync() execution resumes here:
    pin_status = emccanon.GET_EXTERNAL_DIGITAL_INPUT(0,0);
    print("pin status=",pin_status)
---------------------------------------------------------------------

WARNING: The 'yield' feature is fragile. The following restrictions
apply to the usage of 'yield INTERP_EXECUTE_FINISH':

- Python code executing a 'yield INTERP_EXECUTE_FINISH' must be part
  of a remap procedure. Yield does not work in a Python oword procedure.
- A Python remap subroutine containing 'yield INTERP_EXECUTE_FINISH' statement may
  not return a value, as with normal Python yield statements.
- Code following a yield may not recursively call the interpreter, like with
  self.execute("<mdi command>"). This is an architectural restriction
  of the interpreter and is not fixable without a major redesign.

=== Calling conventions: Python to NGC

NGC code is executed from Python when:

- the method `self.execute(<NGC code>[,<line number>])` is executed
- during execution of a remapped code, if a `prolog=` function is
  defined, the NGC procedure given in `ngc=` is executed immediately
  thereafter.

The prolog handler does not call the handler, but it prepares its call
environment, for instance by setting up predefined local parameters.

.Inserting parameters in a prolog, and retrieving  them in an epilog

Conceptually a prolog and an epilog execute at the same call level
like the O-word procedure, that is: after the subroutine call is set
up, and before the subroutine endsub or return.

This means that any local variable created in a prolog will be a local
variable in the O-word procedure, and any local variables created in
the O-word procedure are still accessible when the epilog executes.

The `self.params` array handles reading and setting numbered and named
parameters. If a named parameter begins with `_` (underscore), it is
assumed to be a global parameter; if not, it is local to the calling
procedure. Also, numbered parameters in the range 1..30 are treated
like local variables; their original values are restored on
return/endsub from an O-word procedure.

Here is an example remapped code demonstrating insertion and
extraction of parameters into/from the O-word procedure:

----
REMAP=m300 prolog=insert_param ngc=testparam epilog=retrieve_param modalgroup=10
----

[source,python]
---------------------------------------------------------------------
def insert_param(self, **words): # in the remap module
    print("insert_param call level=",self.call_level)
    self.params["myname"] = 123
    self.params[1] = 345
    self.params[2] = 678
    return INTERP_OK

def retrieve_param(self, **words):
    print("retrieve_param call level=",self.call_level)
    print("#1=", self.params[1])
    print("#2=", self.params[2])
    try:
        print("result=", self.params["result"])
    except Exception,e:
  return "testparam forgot to assign #<result>"
    return INTERP_OK
---------------------------------------------------------------------

[source,{ngc}]
---------------------------------------------------------------------
o<testparam> sub
(debug, call_level=#<_call_level> myname=#<myname>)
; try commenting out the next line and run again
#<result> = [#<myname> * 3]
#1 = [#1 * 5]
#2 = [#2 * 3]
o<testparam> endsub
m2
---------------------------------------------------------------------

`self.params()` returns a list of all variable names currently defined.
Since `myname` is local, it goes away after the epilog finishes.

.Calling the interpreter from Python

You can recursively call the interpreter from Python code as follows:

----
self.execute(<NGC code>[,<line number>])
----

Examples:

[source,python]
---------------------------------------------------------------------
  self.execute("G1 X%f Y%f" % (x,y))
  self.execute("O <myprocedure> call", currentline)
---------------------------------------------------------------------

You might want to test for the return value being `<
INTERP_MIN_ERROR`. If you're using lots of execute() statements, it's
probably easier to trap InterpreterException as per below.

CAUTION: The parameter insertion/retrieval method described in the previous section does not
work in this case. It is good enough for just executing simple NGC
commands or a procedure call and advanced introspection into the
procedure, and passing of local named parameters is not needed. The
recursive call feature is fragile.

.Interpreter Exception during execute()

if `interpreter.throw_exceptions` is nonzero (default 1), and self.execute() returns an error, the exception
`InterpreterException` is raised. InterpreterException has the
following attributes:

`line_number`::
  where the error occurred
`line_text`::
  the NGC statement causing the error
`error_message`::
  the interpreter's error message

Errors can be trapped in the following Pythonic way:

[source,python]
-------------------------------------------------------------------
import interpreter
interpreter.throw_exceptions = 1
   ...
   try:
        self.execute("G3456")  #  raise InterpreterException

   except InterpreterException,e:
        msg = "%d: '%s' - %s" % (e.line_number,e.line_text, e.error_message)
        return msg  # replace builtin error message
-------------------------------------------------------------------

// NOTE: to iterate is human, to recurse: divine.

.Canon

The canon layer is practically all free functions.
Example:

[source,python]
---------------------------------------------------------------------
import emccanon
def example(self,*args):
    ....
    emccanon.STRAIGHT_TRAVERSE(line,x0,y0,z0,0,0,0,0,0,0)
    emccanon.STRAIGHT_FEED(line,x1,y1,z1,0,0,0,0,0,0)
    ...
    return INTERP_OK
---------------------------------------------------------------------

The actual canon functions are declared in `src/emc/nml_intf/canon.hh`
and implemented in `src/emc/task/emccanon.cc`.  The implementation of
the Python functions can be found in `src/emc/rs274ncg/canonmodule.cc`.

=== Built in modules

The following modules are built in:

`interpreter`::
  exposes internals of the Interp class. See
  `src/emc/rs274ngc/interpmodule.cc`, and the
  `tests/remap/introspect` regression test.

`emccanon`::
  exposes most calls of  `src/emc/task/emccanon.cc`.

`emctask`::
  exposes the `emcStatus` class instance. See  `src/emc/task/taskmodule.cc`.
  Not present when using the `gcode` module used for user
  interfaces - only present in the milltask instance of the interpreter.

[[remap:adding-predefined-named-parameters]]
== Adding Predefined Named Parameters

The interpreter comes with a set of predefined named parameters for
accessing internal state from the NGC language level. These parameters
are read-only and global, and hence cannot be assigned to.

Additional parameters may be added by defining a function in the
`namedparams` module. The name of the function defines the name of the
new predefined named parameter, which now can be referenced in
arbitrary expressions.

To add or redefine a named parameter:

* add a `namedparams` module so it can be found by the interpreter
* define new parameters by functions (see below). These functions
  receive `self` (the interpreter instance) as parameter and so can
  access arbitrary state. Arbitrary Python capabilities can be used to return a value.
* import that module from the `TOPLEVEL` script

[source,python]
---------------------------------------------------------------------
# namedparams.py
# trivial example
def _pi(self):
    return 3.1415926535
---------------------------------------------------------------------

[source,{ngc}]
---------------------------------------------------------------------
#<circumference> = [2 * #<radius> * #<_pi>]
---------------------------------------------------------------------

Functions in `namedparams.py` are expected to return a float or int
value. If a string is returned, this sets the interpreter error
message and aborts execution.

Ònly functions with a leading underscore are added as parameters,
since this is the RS274NGC convention for globals.

It is possible to redefine an existing predefined parameter by adding
a Python function of the same name to the `namedparams` module. In
this case, a warning is generated during startup.

While the above example isn't terribly useful, note that pretty much
all of the interpreter internal state is accessible from Python, so
arbitrary predicates may be defined this way. For a slightly more
advanced example, see `tests/remap/predefined-named-params`.

[[remap:standard-glue]]
== Standard Glue routines

Since many remapping tasks are very similar, I've started collecting
working prolog and epilog routines in a single Python module. These
can currently be found in
'ncfiles/remap_lib/python-stdglue/stdglue.py' and provide the
following routines:

=== T: +prepare_prolog+ and +prepare_epilog+

These wrap a NGC procedure for Tx Tool Prepare.

.Actions of +prepare_prolog+

The following parameters are made visible to the NGC procedure:

- `#<tool>` - the parameter of the `T` word
- `#<pocket>` - the corresponding pocket

If tool number zero is requested (meaning Tool unload), the
corresponding pocket is passed as -1.

It is an error if:

- no tool number is given as T parameter
- the tool cannot be found in the tool table.

Note that unless you set the `[EMCIO] RANDOM_TOOLCHANGER=1` parameter,
tool and pocket number are identical, and the pocket number from the
tool table is ignored. This is currently a restriction.

.Actions of +prepare_epilog+

- The NGC procedure is expected to return a positive value, otherwise
  and error message containing the return value is given and the
  interpreter aborts.
- In case the NGC procedure executed the T command (which then refers
  to the built in T behavior), no further action is taken. This can be
  used for instance to minimally adjust the built in behavior be
  preceding or following it with some other statements.
- Otherwise, the `#<tool>` and `#<pocket>` parameters are extracted
  from the subroutine's parameter space. This means that the NGC
  procedure could change these values, and the epilog takes the
  changed values in account.
- then, the Canon command `SELECT_TOOL(#<tool>)` is executed.

=== M6: +change_prolog+ and +change_epilog+

These wrap a NGC procedure for M6 Tool Change.

.Actions of +change_prolog+

* The following three steps are applicable only if the `iocontrol-v2` component is used:
** If parameter 5600 (fault indicator) is greater than zero, this indicates a Toolchanger
   fault, which is handled as follows:
** if parameter 5601 (error code) is negative, this indicates a hard
   fault and the prolog aborts with an error message.
** if parameter 5601 (error code) is greater equal zero, this indicates a soft
   fault. An informational message is displayed and the prolog continues.
* If there was no preceding T command which caused a pocket to be
  selected, the prolog aborts with an error message.
* If cutter radius compensation is on, the prolog aborts with an error
  message.

Then, the following parameters are exported to the NGC procedure:

- `#<tool_in_spindle>` : the tool number of the currently loaded tool
- `#<selected_tool>` : the tool number selected
- `#<selected_pocket>` : the selected tool's tooldata index

.Actions of +change_epilog

* The NGC procedure is expected to return a positive value, otherwise
  and error message containing the return value is given and the
  interpreter aborts.
* If parameter 5600 (fault indicator) is greater than zero, this indicates a Toolchanger
  fault, which is handled as follows (`iocontrol-v2`-only):
** if parameter 5601 (error code) is negative, this indicates a hard
   fault and the epilog aborts with an error message.
** if parameter 5601 (error code) is greater equal zero, this indicates a soft
   fault. An informational message is displayed and the epilog continues.
* In case the NGC procedure executed the M6 command (which then refers
  to the built in M6 behavior), no further action is taken. This can be
  used for instance to minimally adjust the built in behavior be
  preceding or following it with some other statements.
* Otherwise, the `#<selected_pocket>` parameter is extracted
  from the subroutine's parameter space, and used to set the
  interpreter's `current_pocket` variable. Again, the
  procedure could change this value, and the epilog takes the
  changed value in account.
* then, the Canon command `CHANGE_TOOL(#<selected_pocket>)` is
  executed.
* The new tool parameters (offsets, diameter etc) are set.

[[remap:cycle-stdglue]]
=== G-code Cycles: +cycle_prolog+ and +cycle_epilog+

These wrap a NGC procedure so it can act as a cycle, meaning the
motion code is retained after finishing execution. If the next line
just contains parameter words (e.g. new X,Y values), the code is
executed again with the new parameter words merged into the set of the
parameters given in the first invocation.

These routines are designed to work in conjunction with an
<<sub:argspec-parameter,`argspec=<words>` parameter>>. While this is
easy to use, in a realistic scenario you would avoid argspec and do a
more thorough investigation of the block manually in order to give better
error messages.

The suggested argspec is as follows:

[source,{ini}]
---------------------------------------------------------------------
REMAP=G<somecode> argspec=xyzabcuvwqplr prolog=cycle_prolog ngc=<ngc procedure> epilog=cycle_epilog modalgroup=1
---------------------------------------------------------------------

This will permit +cycle_prolog+ to determine the compatibility of any
axis words give in the block, see below.

.Actions of +cycle_prolog+
* Determine whether the words passed in from the current block fulfill
  the conditions outlined under <<gcode:canned-cycle-errors,Canned Cycle Errors>>.
** export the axis words as +<x>+, +#<y>+ etc; fail if axis words from
   different groups (XYZ) (UVW) are used together, or any of (ABC) is given.
** export 'L-' as +#<l>+; default to 1 if not given.
** export 'P-' as +#<p>+; fail if p less than 0.
** export 'R-' as +#<r>+; fail if r not given, or less equal 0 if given.
** fail if feed rate is zero, or inverse time feed or cutter
   compensation is on.
* Determine whether this is the first invocation of a cycle G-code, if so:
** Add the words passed in (as per argspec) into a set of sticky
   parameters, which is retained across several invocations.
* If not (a continuation line with new parameters):
** merge the words passed in into the existing set of sticky parameters.
* export the set of sticky parameters to the NGC procedure.

.Actions of +cycle_epilog+
* Determine if the current code was in fact a cycle, if so:
** retain the current motion mode so a continuation line without a
   motion code will execute the same motion code.

=== S (Set Speed) : +setspeed_prolog+ and +setspeed_epilog+

TBD

=== F (Set Feed) : +setfeed_prolog+ and +setfeed_epilog+

TBD

=== M61 Set tool number : +settool_prolog+ and +settool_epilog+

TBD

== Remapped code execution

=== NGC procedure call environment during remaps

Normally, an O-word procedure is called with positional parameters.
This scheme is very limiting in particular in the presence of optional
parameters. Therefore, the calling convention has been extended to use
something remotely similar to the Python keyword arguments model.

see  LINKTO gcode/main Subroutines: sub, endsub, return, call.

=== Nested remapped codes

Remapped codes may be nested just like procedure calls - that is, a
remapped code whose NGC procedure refers to some other remapped code
will execute properly.

The maximum nesting level remaps is currently 10.

=== Sequence number  during remaps

Sequence numbers are propagated and restored like with O-word
calls. See `tests/remap/nested-remaps/word` for the regression test,
which shows sequence number tracking during nested remaps three levels
deep.

=== Debugging flags

The following flags are relevant for remapping and Python - related
execution:

  EMC_DEBUG_OWORD             0x00002000  traces execution of O-word subroutines
  EMC_DEBUG_REMAP             0x00004000  traces execution of remap-related code
  EMC_DEBUG_PYTHON            0x00008000  calls to the Python plug in
  EMC_DEBUG_NAMEDPARAM        0x00010000  trace named parameter access
  EMC_DEBUG_PYTHON_TASK       0x00040000  trace the task Python plug in
  EMC_DEBUG_USER1             0x10000000  user-defined - not interpreted by LinuxCNC
  EMC_DEBUG_USER2             0x20000000  user-defined - not interpreted by LinuxCNC

'or' these flags into the `[EMC]DEBUG` variable as needed. For a current
list of debug flags see 'src/emc/nml_intf/debugflags.h'.

=== Debugging Embedded Python code

Debugging of embedded Python code is harder than debugging normal
Python scripts, and only a limited supply of debuggers exists. A
working open-source based solution is to use the
http://www.eclipse.org[Eclipse IDE], and the http://www.pydev.org[PydDev]
Eclipse plug in and its
http://pydev.org/manual_adv_remote_debugger.html[remote debugging
feature].

To use this approach:

- install Eclipse via the the 'Ubuntu Software Center' (choose first selection)
- install the PyDev plug in from the http://pydev.org/updates[Pydev Update Site]
- setup the LinuxCNC source tree as an Eclipse project
- start the Pydev Debug Server in Eclipse
- make sure the embedded Python code can find the `pydevd.py` module
  which comes with that plug in - it's buried somewhere deep under the
  Eclipse install directory. Set the the `pydevd` variable in `util.py`
  to reflect this directory location.
- `import pydevd` in your Python module - see example `util.py` and `remap.py`
- call `pydevd.settrace()` in your module at some point to connect to
  the Eclipse Python debug server - here you can set breakpoints in your
  code, inspect variables, step etc as usual.

CAUTION: `pydevd.settrace()` will block execution if Eclipse and the
Pydev debug server have not been started.

To cover the last two steps: the `o<pydevd>` procedure helps to get
into the debugger from MDI mode. See also the `call_pydevd` function
in `util.py` and its usage in `remap.involute` to set a breakpoint.

Here's a screen-shot of Eclipse/PyDevd debugging the `involute`
procedure from above:

image::images/debug_embedded_python.png[Debugging with Eclipse]

See the Python code in `configs/sim/axis/remap/getting-started/python` for details.

[[remap:axis-preview-and-remapped-code-execution]]
== Axis Preview and Remapped code execution

For complete preview of a remapped code's tool path some precautions
need to be taken. To understand what is going on, let's review the
preview and execution process (this covers the Axis case, but others
are similar):

First, note that there are *two* independent interpreter instances
involved:

- one instance in the milltask program, which executes a program when
  you hit the 'Start' button, and actually makes the machine move
- a second instance in the user interface whose primary purpose is to
  generate the tool path preview. This one 'executes' a program once it
  is loaded, but it doesn't actually cause machine movements.

Now assume that your remap procedure contains a G38 probe operation,
for example as part of a tool change with automatic tool length
touch off. If the probe fails, that would clearly be an error, so you'd
display a message and abort the program.

Now, what about preview of this procedure?  At preview time, of course
it's not known whether the probe succeeds or fails - but you would
likely want to see what the maximum depth of the probe is, and assume
it  succeeds and continues execution to
preview further movements. Also, there is no point in displaying a
'probe failed' message and aborting *during preview*.

The way to address this issue is to test in your procedure whether it
executes in preview or execution mode. This can be checked for by
testing the `#<_task>` <<gcode:predefined-named-parameters,predefined
named parameter>> - it will be 1 during actual execution and 0 during
preview. See
'configs/sim/axis/remap/manual-toolchange-with-tool-length-switch/nc_subroutines/manual_change.ngc'
for a complete usage example.

Within Embedded Python, the `task` instance can be checked for by
testing 'self.task' - this will be 1 in the milltask instance, and 0
in the preview instance(s).

[[remap:remappable-codes]]
== Remappable Codes

[[remap:existing-codes]]
=== Existing codes which can be remapped

The current set of *existing* codes open to redefinition is:

- Tx (Prepare)
- M6 (Change tool)
- M61 (Set tool number)
- M0 (pause a running program temporarily)
- M1 (pause a running program temporarily if the optional stop switch is on)
- M60 (exchange pallet shuttles and then pause a running program temporarily)
- S  (set spindle speed)
- F  (set feed)

Note that the use of M61 currently requires the use of `iocontrol-v2`.

[[remap:unallocated-g-codes]]
=== Currently unallocated G-codes:

Currently unallocated G-codes (for remapping) must be selected from the blank
areas of the following tables. All the listed G-codes are already defined in
the current implementation of LinuxCNC and may *not* be used to remap new
G-codes. (Developers who add new G-codes to LinuxCNC are encouraged to also
add their new G-codes to these tables.)

.Table of Allocated G-codes 00-09
[width="90%",align="center",options="header,strong,unbreakable",cols="1*2^em,10*1<m"]
|==============================================================
|#  |Gxx |Gxx.1|Gxx.2|Gxx.3|Gxx.4|Gxx.5|Gxx.6|Gxx.7|Gxx.8|Gxx.9
|00 |G00 |     |     |     |     |     |     |     |     | 
|01 |G01 |     |     |     |     |     |     |     |     | 
|02 |G02 |     |     |     |     |     |     |     |     | 
|03 |G03 |     |     |     |     |     |     |     |     | 
|04 |G04 |     |     |     |     |     |     |     |     | 
|05 |G05 |G05.1|G05.2|G05.3|     |     |     |     |     | 
|06 |    |     |     |     |     |     |     |     |     | 
|07 |G07 |     |     |     |     |     |     |     |     | 
|08 |G08 |     |     |     |     |     |     |     |     | 
|09 |    |     |     |     |     |     |     |     |     | 
|==============================================================

.Table of Allocated G-codes 10-19
[width="90%",align="center",options="header,strong,unbreakable",cols="1*2^em,10*1<m"]
|==============================================================
|#  |Gxx |Gxx.1|Gxx.2|Gxx.3|Gxx.4|Gxx.5|Gxx.6|Gxx.7|Gxx.8|Gxx.9
|10 |G10 |     |     |     |     |     |     |     |     | 
|11 |    |     |     |     |     |     |     |     |     | 
|12 |    |     |     |     |     |     |     |     |     | 
|13 |    |     |     |     |     |     |     |     |     | 
|14 |    |     |     |     |     |     |     |     |     | 
|15 |    |     |     |     |     |     |     |     |     | 
|16 |    |     |     |     |     |     |     |     |     | 
|17 |G17 |G17.1|     |     |     |     |     |     |     | 
|18 |G18 |G18.1|     |     |     |     |     |     |     | 
|19 |G19 |G19.1|     |     |     |     |     |     |     | 
|==============================================================

.Table of Allocated G-codes 20-29
[width="90%",align="center",options="header,strong,unbreakable",cols="1*2^em,10*1<m"]
|==============================================================
|#  |Gxx |Gxx.1|Gxx.2|Gxx.3|Gxx.4|Gxx.5|Gxx.6|Gxx.7|Gxx.8|Gxx.9
|20 |G20 |     |     |     |     |     |     |     |     | 
|21 |G21 |     |     |     |     |     |     |     |     | 
|22 |    |     |     |     |     |     |     |     |     | 
|23 |    |     |     |     |     |     |     |     |     | 
|24 |    |     |     |     |     |     |     |     |     | 
|25 |    |     |     |     |     |     |     |     |     | 
|26 |    |     |     |     |     |     |     |     |     | 
|27 |    |     |     |     |     |     |     |     |     | 
|28 |G28 |G28.1|     |     |     |     |     |     |     | 
|29 |    |     |     |     |     |     |     |     |     | 
|==============================================================

.Table of Allocated G-codes 30-39
[width="90%",align="center",options="header,strong,unbreakable",cols="1*2^em,10*1<m"]
|==============================================================
|#  |Gxx |Gxx.1|Gxx.2|Gxx.3|Gxx.4|Gxx.5|Gxx.6|Gxx.7|Gxx.8|Gxx.9
|30 |G30 |G30.1|     |     |     |     |     |     |     | 
|31 |    |     |     |     |     |     |     |     |     | 
|32 |    |     |     |     |     |     |     |     |     | 
|33 |G30 |G30.1|     |     |     |     |     |     |     | 
|34 |    |     |     |     |     |     |     |     |     | 
|35 |    |     |     |     |     |     |     |     |     | 
|36 |    |     |     |     |     |     |     |     |     | 
|37 |    |     |     |     |     |     |     |     |     | 
|38 |    |     |     |     |     |     |     |     |     | 
|39 |    |     |     |     |     |     |     |     |     | 
|==============================================================

.Table of Allocated G-codes 40-49
[width="90%",align="center",options="header,strong,unbreakable",cols="1*2^em,10*1<m"]
|==============================================================
|#  |Gxx |Gxx.1|Gxx.2|Gxx.3|Gxx.4|Gxx.5|Gxx.6|Gxx.7|Gxx.8|Gxx.9
|40 |G40 |     |     |     |     |     |     |     |     | 
|41 |G41 |G41.1|     |     |     |     |     |     |     | 
|42 |G42 |G42.1|     |     |     |     |     |     |     | 
|43 |G43 |G43.1|     |     |     |     |     |     |     | 
|44 |    |     |     |     |     |     |     |     |     | 
|45 |    |     |     |     |     |     |     |     |     | 
|46 |    |     |     |     |     |     |     |     |     | 
|47 |    |     |     |     |     |     |     |     |     | 
|48 |    |     |     |     |     |     |     |     |     | 
|49 |G40 |     |     |     |     |     |     |     |     | 
|==============================================================

.Table of Allocated G-codes 50-59
[width="90%",align="center",options="header,strong,unbreakable",cols="1*2^em,10*1<m"]
|==============================================================
|#  |Gxx |Gxx.1|Gxx.2|Gxx.3|Gxx.4|Gxx.5|Gxx.6|Gxx.7|Gxx.8|Gxx.9
|50 |    |     |     |     |     |     |     |     |     | 
|51 |    |     |     |     |     |     |     |     |     | 
|52 |    |     |     |     |     |     |     |     |     | 
|53 |G53 |     |     |     |     |     |     |     |     | 
|54 |G54 |     |     |     |     |     |     |     |     | 
|55 |G55 |     |     |     |     |     |     |     |     | 
|56 |G56 |     |     |     |     |     |     |     |     | 
|57 |G57 |     |     |     |     |     |     |     |     | 
|58 |G58 |     |     |     |     |     |     |     |     | 
|59 |G59 |G59.1|G59.2|G59.3|     |     |     |     |     | 
|==============================================================

.Table of Allocated G-codes 60-69
[width="90%",align="center",options="header,strong,unbreakable",cols="1*2^em,10*1<m"]
|==============================================================
|#  |Gxx |Gxx.1|Gxx.2|Gxx.3|Gxx.4|Gxx.5|Gxx.6|Gxx.7|Gxx.8|Gxx.9
|60 |G60 |     |     |     |     |     |     |     |     | 
|61 |G61 |G61.1|     |     |     |     |     |     |     | 
|62 |    |     |     |     |     |     |     |     |     | 
|63 |    |     |     |     |     |     |     |     |     | 
|64 |G64 |     |     |     |     |     |     |     |     | 
|65 |    |     |     |     |     |     |     |     |     | 
|66 |    |     |     |     |     |     |     |     |     | 
|67 |    |     |     |     |     |     |     |     |     | 
|68 |    |     |     |     |     |     |     |     |     | 
|69 |    |     |     |     |     |     |     |     |     | 
|==============================================================

.Table of Allocated G-codes 70-79
[width="90%",align="center",options="header,strong,unbreakable",cols="1*2^em,10*1<m"]
|==============================================================
|#  |Gxx |Gxx.1|Gxx.2|Gxx.3|Gxx.4|Gxx.5|Gxx.6|Gxx.7|Gxx.8|Gxx.9
|70 |G70 |     |     |     |     |     |     |     |     | 
|71 |G71 |G71.1|G71.2|     |     |     |     |     |     | 
|72 |G72 |G72.1|G72.2|     |     |     |     |     |     | 
|73 |    |     |     |     |     |     |     |     |     | 
|74 |    |     |     |     |     |     |     |     |     | 
|75 |    |     |     |     |     |     |     |     |     | 
|76 |G76 |     |     |     |     |     |     |     |     | 
|77 |    |     |     |     |     |     |     |     |     | 
|78 |    |     |     |     |     |     |     |     |     | 
|79 |    |     |     |     |     |     |     |     |     | 
|==============================================================

.Table of Allocated G-codes 80-89
[width="90%",align="center",options="header,strong,unbreakable",cols="1*2^em,10*1<m"]
|==============================================================
|#  |Gxx |Gxx.1|Gxx.2|Gxx.3|Gxx.4|Gxx.5|Gxx.6|Gxx.7|Gxx.8|Gxx.9
|80 |G80 |     |     |     |     |     |     |     |     | 
|81 |G81 |     |     |     |     |     |     |     |     | 
|82 |G82 |     |     |     |     |     |     |     |     | 
|83 |G83 |     |     |     |     |     |     |     |     | 
|84 |G84 |     |     |     |     |     |     |     |     | 
|85 |G85 |     |     |     |     |     |     |     |     | 
|86 |G86 |     |     |     |     |     |     |     |     | 
|87 |G87 |     |     |     |     |     |     |     |     | 
|88 |G88 |     |     |     |     |     |     |     |     | 
|89 |G89 |     |     |     |     |     |     |     |     | 
|==============================================================

.Table of Allocated G-codes 90-99
[width="90%",align="center",options="header,strong,unbreakable",cols="1*2^em,10*1<m"]
|==============================================================
|#  |Gxx |Gxx.1|Gxx.2|Gxx.3|Gxx.4|Gxx.5|Gxx.6|Gxx.7|Gxx.8|Gxx.9
|90 |G90 |G90.1|     |     |     |     |     |     |     | 
|91 |G91 |G91.1|     |     |     |     |     |     |     | 
|92 |G92 |G92.1|G92.2|G92.3|     |     |     |     |     | 
|93 |G93 |     |     |     |     |     |     |     |     | 
|94 |G94 |     |     |     |     |     |     |     |     | 
|95 |G95 |     |     |     |     |     |     |     |     | 
|96 |G96 |     |     |     |     |     |     |     |     | 
|97 |G97 |     |     |     |     |     |     |     |     | 
|98 |G98 |     |     |     |     |     |     |     |     | 
|99 |G99 |     |     |     |     |     |     |     |     | 
|==============================================================

[[remap:unallocated-m-codes]]
=== Currently unallocated M-codes:

These M-codes are currently undefined in the current implementation of LinuxCNC
and may be used to define new M-codes. (Developers who define new M-codes in
LinuxCNC are encouraged to remove them from this table.)

.Table of Unallocated M-codes 00-99
[width="90%",align="center",options="header,strong,unbreakable",cols="1*2^em,10*1<m"]
|==============================================================
|#     |Mx0 |Mx1 |Mx2 |Mx3 |Mx4 |Mx5 |Mx6 |Mx7 |Mx8 |Mx9
|00-09 |    |    |    |    |    |    |    |    |    | 
|10-19 |M10 |M11 |M12 |M13 |M14 |M15 |M16 |M17 |M18 | 
|20-29 |M20 |M21 |M22 |M23 |M24 |M25 |M26 |M27 |M28 |M29
|30-39 |    |M31 |M32 |M33 |M34 |M35 |M36 |M37 |M38 |M39
|40-49 |M40 |M41 |M42 |M43 |M44 |M45 |M46 |M47 |    | 
|50-59 |    |    |    |    |M54 |M55 |M56 |M57 |M58 |M59
|60-69 |    |    |    |    |    |    |    |    |    | 
|70-79 |    |    |    |    |M74 |M75 |M76 |M77 |M78 |M79
|80-89 |M80 |M81 |M82 |M83 |M84 |M85 |M86 |M87 |M88 |M89
|90-99 |M90 |M91 |M92 |M93 |M94 |M95 |M96 |M97 |M98 |M99
|==============================================================

All M-codes from `M100` to `M199` are user-defined M-codes already,
and should not be remapped.

All M-codes from `M200` to `M999` are available for remapping.

=== readahead time and execution time

FIXME Write missing information

=== plugin/pickle hack

FIXME Write missing information

=== Module, methods, classes, etc reference

FIXME Write missing information

== Introduction: Extending Task Execution

FIXME Write missing information

=== Why would you want to change Task Execution?

FIXME Write missing information

=== A diagram: `task`, `interp`, `iocontrol`, `UI` (??)

FIXME Write missing information

== Models of Task execution

FIXME Write missing information

=== Traditional `iocontrol`/`iocontrolv2` execution

FIXME Write missing information

=== Redefining IO procedures

FIXME Write missing information

=== Execution-time Python procedures

FIXME Write missing information

// setup examples

== A short survey of LinuxCNC program execution

To understand remapping of codes it might be helpful to survey the
execution of `task` and interpreter as far as it relates to remapping.

=== Interpreter state

Conceptually, the interpreter's state consist of variables which fall into
the following categories:

1. _Configuration information_ (typically from INI file)
2. _The 'World model'_  - a representation of actual machine state
3. _Modal state and settings_ -
   Refers to state which is 'carried over' between executing
   individual NGC codes - for instance, once the spindle is turned on and
   the speed is set, it remains at this setting until turned off. The
   same goes for many codes, like feed, units, motion modes (feed or
   rapid)  and so forth.
4. _Interpreter execution state_ -
   Holds information about the block currently executed, whether we
   are in a subroutine, interpreter variables, etc. .
   Most of this state is aggregated in a - fairly unsystematic -
   `structure _setup` (see interp_internals.hh).

=== Task and Interpreter interaction, Queuing and Read-Ahead

The `task` part of LinuxCNC is responsible for coordinating actual machine
commands - movement, HAL interactions and so forth. It does not by
itself handle the RS274NGC language. To do so, `task` calls upon the
interpreter to parse and execute the next command - either from MDI or
the current file.

The interpreter execution generates canonical machine operations, which
actually move something. These are, however, not immediately executed but
put on a queue. The actual execution of these codes happens in the `task`
part of LinuxCNC: canon commands are pulled off that interpreter queue,
and executed resulting in actual machine movements.

This means that typically the interpreter is far ahead of the actual
execution of commands - the parsing of the program might well be
finished before any noticeable movement starts. This behavior is
called 'read-ahead'.

=== Predicting the machine position

To compute canonical machine operations in advance during read ahead,
the interpreter must be able to predict the machine position after
each line of G-code, and that is not always possible.

Let's look at a simple example program which does relative moves
(G91), and assume the machine starts at x=0,y=0,z=0. Relative moves
imply that the outcome of the next move relies on the position of the
previous one:

[source,{ngc}]
---------------------------------------------------------------------
N10 G91
N20 G0 X10 Y-5 Z20
N30 G1 Y20 Z-5
N40 G0 Z30
N50 M2
---------------------------------------------------------------------

Here the interpreter can clearly predict machine positions for each line:

After N20: x=10 y=-5 z=20; after N30: x=10 y=15 z=15; after N40: x=10 y=15 z=45

and so can parse the whole program and generate canonical operations
well in advance.

=== Queue-busters break position prediction

However, complete read ahead is only possible when the
interpreter can predict the position impact for *every* line in the
program in advance. Let's look at a modified example:

[source,{ngc}]
---------------------------------------------------------------------
N10 G91
N20 G0 X10 Y-5 Z20
N30 G38.3 Z-10
N40 O100 if [#5070 EQ 0]
N50    G1 Y20 Z-5
N60 O100 else
N70    G0 Z30
N80 O100 endif
N90 G1 Z10
N95 M2
---------------------------------------------------------------------

To pre-compute the move in N90, the interpreter would need to know
where the machine is after line N80 - and that depends on whether the
probe command succeeded or not, which is not known until it's actually
executed.

So, some operations are incompatible with further read-ahead. These
are called 'queue busters', and they are:

- Reading a HAL pin's value with M66: value of HAL pin not predictable.
- Loading a new tool with M6: tool geometry not predictable.
- Executing a probe with G38.n: final position and success/failure not predictable.

=== How queue-busters are dealt with

Whenever the interpreter encounters a queue-buster, it needs to stop
read ahead and wait until the relevant result is available. The way
this works is:

- When such a code is encountered, the interpreter returns a
  special return code to `task` ('INTERP_EXECUTE_FINISH').
- This return code signals to `task` to stop read ahead for now, execute
  all queued canonical commands built up so far (including the last one,
  which is the queue buster), and then 'synchronize
  the interpreter state with the world model'. Technically, this means
  updating internal variables to reflect HAL pin values, reload tool
  geometries after an M6, and convey results of a probe.
- The interpreter's 'synch()' method is called by `task` and does just
  that - read all the world model 'actual' values which are relevant for
  further execution.
- At this point, `task` goes ahead and calls the interpreter for more
  read ahead - until either the program ends or another queue-buster is encountered.

=== Word order and execution order

One or several 'words' may be present on an NGC 'block' if they are
compatible (some are mutually exclusive and must be on different
lines).  The execution model however prescribes a strict ordering of
execution of codes, regardless of their appearance on the source line
(<<gcode:order-of-execution,G-code Order of Execution>>).

=== Parsing

Once a line is read (in either MDI mode, or from the current NGC
file), it is parsed and flags and parameters are set in a 'struct
block' (struct _setup, member block1). This struct holds all information
about the current source line, but independent of different ordering
of codes on the current line: as long as several codes are compatible,
any source ordering will result in the same variables set in the
struct block. Right after parsing, all codes on a block are checked for
compatibility.

=== Execution

After successful parsing the block is executed by execute_block(), and
here the different items are handled according to execution order.

If a "queue buster" is found, a corresponding flag is set in the
interpreter state (toolchange_flag, input_flag, probe_flag) and the
interpreter returns an INTERP_EXECUTE_FINISH return value, signaling
'stop readahead for now, and resynch' to the caller ('`task`').
If no queue busters are found
after all items are executed, INTERP_OK is returned, signalling that
read-ahead may continue.

When read ahead continues after the synch, `task` starts executing
interpreter read() operations again.  During the next read operation,
the above mentioned flags are checked and corresponding variables are
set (because the a synch() was just executed, the values are now
current). This means that the next command already executes in the
properly set variable context.

=== Procedure execution

O-word procedures complicate handling of queue busters a bit. A queue
buster might be found somewhere in a nested procedure, resulting in a
semi-finished procedure call when INTERP_EXECUTE_FINISH is
returned. Task makes sure to synchronize the world model, and continue
parsing and execution as long as there is still a procedure executing
(call_level > 0).

[[remap:how-tool-change-currently-works]]
=== How tool change currently works

The actions happening in LinuxCNC are a bit involved, but it's necessary
to get the overall idea what currently happens before you set out to
adapt those workings to your own needs.

Note that remapping an existing code completely disables all internal
processing for that code. That means that beyond your desired
behavior - probably described through an NGC Oword or Python
procedure, you need to replicate those internal actions of the
interpreter which together result in a complete replacement of the
existing code. The prolog and epilog code is the place to do this.

.How tool information is communicated
Several processes are 'interested' in tool information: `task` and its
interpreter, as well as the user interface. Also, the 'halui' process.

Tool information is held in the 'emcStatus' structure, which is
shared by all parties. One of its fields is the 'toolTable' array,
which holds the description as loaded from the tool table file (tool
number, diameter, frontangle, backangle and orientation for lathe,
tool offset information).

The authoritative source and only process actually 'setting' tool
information in this structure is the 'iocontrol' process. All others
processes just consult this structure. The interpreter holds actually
a local copy of the tool table.

For the curious, the current emcStatus structure can be accessed by
<<remap:executing-python-statements,Python statements>>. The
interpreter's perception of the tool currently loaded for instance is
accessed by:

----
;py,from interpreter import *
;py,print(this.tool_table[0])
----

To see fields in the global emcStatus structure, try this:

----
;py,from emctask import *
;py,print(emcstat.io.tool.pocketPrepped)
;py,print(emcstat.io.tool.toolInSpindle)
;py,print(emcstat.io.tool.toolTable[0])
----

You need to have LinuxCNC started from a terminal window to see the
results.

=== How Tx (Prepare Tool) works

[[remap:interpreter-action-on-t]]
.Interpreter action on a Tx command
All the interpreter does is evaluate the toolnumber parameter, looks up
its corresponding tooldata index, remembers it in the `selected_pocket`
variable for later, and queues a canon command (SELECT_TOOL).
See 'Interp::convert_tool_select' in 'src/emc/rs274/interp_execute.cc'.


.Task action on SELECT_TOOL
When `task` gets around to handle a SELECT_TOOL, it sends a
EMC_TOOL_PREPARE message to the `iocontrol` process, which handles most
tool-related actions in LinuxCNC.

In the current implementation, `task` actually waits for `iocontrol` to
complete the changer positioning operation, which is not necessary IMO
since it defeats the idea that changer preparation and code execution can
proceed in parallel.

.Iocontrol action on EMC_TOOL_PREPARE
When `iocontrol` sees the select pocket command, it does the related HAL
pin wiggling - it sets the "tool-prep-number" pin to indicate which
tool is next, raises the "tool-prepare" pin, and waits for the
"tool-prepared" pin to go high.

When the changer responds by asserting "tool-prepared", it considers
the prepare phase to be completed and signals `task` to
continue. (again, this 'wait' istn strictly necessary IMO)

.Building the prolog and epilog for Tx
See the Python functions `prepare_prolog` and `prepare_epilog` in
`configs/sim/axis/remap/toolchange/python/toolchange.py`.

=== How M6 (Change tool) works

You need to understand this fully before you can adapt it. It is very
relevant to writing a prolog and epilog handler for a remapped
M6. Remapping an existing codes means you disable the internal steps
taken normally, and replicate them as far as needed for your own
purposes.

Even if you are not familiar with C, I suggest you look at the
'Interp::convert_tool_change' code in
'src/emc/rs274/interp_convert.cc'.

[[remap:interpreter-action-on-m6]]
.Interpreter action on a M6 command

When the interpreter sees an M6, it:

[[remap:send-tool-load-msg]]
. checks whether a T command has already been executed (test
  'settings->selected_pocket' to be >= 0) and fail with 'Need tool
  prepared -Txx- for toolchange' message if not.
. check for cutter compensation being active, and fail with 'Cannot
  change tools with cutter radius compensation on' if so.
. stop the spindle except if the "TOOL_CHANGE_WITH_SPINDLE_ON" INI
  option is set.
. generate a rapid 'Z up' move if if the "TOOL_CHANGE_QUILL_UP" INI
  option is set.
. if TOOL_CHANGE_AT_G30 was set:
.. move the A, B and C indexers if applicable
.. generate rapid move to the G30 position
. execute a CHANGE_TOOL canon command,with the selected pocket as
  parameter. CHANGE_TOOL will:
.. generate a rapid move to TOOL_CHANGE_POSITION if so set in INI
.. enqueue an EMC_TOOL_LOAD NML message to `task`.
. set the numberer parameters 5400-5413 according to the new tool
. signal to `task` to stop calling the interpreter for readahead by
  returning INTERP_EXECUTE_FINISH since M6 is a queue buster.

.What `task` does when it sees a CHANGE_TOOL command
Again, not much more than passing the buck to `iocontrol` by sending it
an EMC_TOOL_LOAD message, and waiting until `iocontrol` has done its
thing.

[[remap:iocontrol-action-on-load]]
.Iocontrol action on EMC_TOOL_LOAD
. it asserts the "tool-change" pin
. it waits for the "tool-changed" pin to become active
. when that has happened:
.. deassert "tool-change"
.. set "tool-prep-number" and "tool-prep-pocket" pins to zero
.. execute the 'load_tool()' function with the pocket as parameter.

The last step actually sets the tooltable entries in the 'emcStatus' structure.
The actual action taken depends on whether the
RANDOM_TOOLCHANGER INI option was set, but at the end of the process
'toolTable[0]' reflects the tool currently in the spindle.

When that has happened:

. `iocontrol` signals `task` to go ahead.
. `task` tells the interpreter to execute a 'synch()' operation, to see what has changed.
. The interpreter 'synch()' pulls all information from the world model needed, among it the changed tool table.

From there on, the interpreter has complete knowledge of the world model and continues with read ahead.

.Building the prolog and epilog for M6
See the Python functions `change_prolog` and `change_epilog` in
`configs/sim/axis/remap/toolchange/python/toolchange.py`.

=== How M61 (Change tool number) works

M61 requires a non-negative `Q`parameter (tool number).
If zero, this means 'unload tool', else 'set current tool number to Q'.

.Building the replacement for M61
An example Python redefinition for M61 can be found in the
`set_tool_number` function in
`configs/sim/axis/remap/toolchange/python/toolchange.py`.


==  Status

. The RELOAD_ON_CHANGE feature is fairly broken.
  Restart after changing a Python file.
. M61 (remapped or not) is broken in `iocontrol` and requires `iocontrol-v2` to actually work.

// A short survey of LinuxCNC execution
// Interpreter internals access - Python view
// Task internals access - Python view

== Changes

- the method to return error messages and fail used to be
  'self.set_errormsg(text)' followed by 'return INTERP_ERROR'.
  This has been replaced by merely returning a string from a Python handler or oword subroutine.
  This sets the error message and aborts the program. Previously there was no clean way to abort a Python oword
  subroutine.

== Debugging

In the '[EMC]' section of the INI file the 'DEBUG' parameter can be changed to
get various levels of debug messages when LinuxCNC is started from a terminal.

----
Debug level, 0 means no messages. See src/emc/nml_intf/debugflags.h for others
DEBUG = 0x00000002 # configuration
DEBUG = 0x7FFFDEFF # no interp,oword
DEBUG = 0x00008000 # py only
DEBUG = 0x0000E000 # py + remap + Oword
DEBUG = 0x0000C002 # py + remap + config
DEBUG = 0x0000C100 # py + remap + Interpreter
DEBUG = 0x0000C140 # py + remap + Interpreter + NML msgs
DEBUG = 0x0000C040 # py + remap + NML
DEBUG = 0x0003E100 # py + remap + Interpreter + oword + signals + namedparams
DEBUG = 0x10000000 # EMC_DEBUG_USER1 - trace statements
DEBUG = 0x20000000 # EMC_DEBUG_USER2 - trap into Python debugger
DEBUG = 0x10008000 # USER1, PYTHON
DEBUG = 0x30008000 # USER1,USER2, PYTHON # USER2 will cause involute to try to connect to pydev
DEBUG = 0x7FFFFFFF # All debug messages
----

// vim: set syntax=asciidoc:
