1. General

1.1. Objective target, special properties

BiDiB is designed to control model railways, as there are control of locos, accessories and safe transmission
from feedback information out of the layout to a connected PC. BiDiB is short for Bidirectional Bus and has
the following advantages:

Automatic Plug & Play without programming in order to avoid address conflicts in modular and/or variable designs.
Switchboard files can be easily copied.

Largely free arrangement of detectors and output devices, no displacement of addresses through new
added blocks.

Full support of BiDiB feedback detectors with transmission of addresses, CV-messages and actual loco speed.

Notification of multiple locos in one section.

Mixed operation of normal and BiDi-ready feedback modules without problems: There is the possibility to
install BiDi detectors at critical points (e.g. single track) and simple, normal detectors at all other sections.

Flexible, open protocol, future-proof.

The above picture shows a typical arrangement. The protocol is suitable for different feedback systems and can manage
simple home layouts up to large exhibition layouts. The Protocol can be easily adapted to nearly all requirements.
For this purpose, there is a mandatory set of query and setting parameters existing that must be implemented in every
feedback system. The following used terms will be summarized and explaned in the glossary.

1.2. Origin and History

Lenz and Tams introduced BiDi feedback systems by 2009, therefore the use of bi-directional data transfer between
DCC decoders and control units was theoretically possible. Suitable feedback systems has been announced, but by the
end of 2010, only the simple Tams RC Talk was available in the market. From the discussion about a future proof
successor, the BiDiB protocol was created. This was the base for an enhanced draft, which also includes command
stations, accessories and control panels.

At this point, I would like to say "Thank you" to my discussion partners, Mr. Tams, Mr. Bluecher and Mr. Herzog
for their constructive input.

1.3. Disclaimer, License

BiDiB® is maintained and developed further by a working group (BiDiB-Implement-Group). Among other, the sirs
Kersten Tams, Markus Herzog and Wolfgang Kufer are authoritative members of this group.

Legal: This specification is provided "as is" and without any warranty of any kind, expressed or implied.
Without limitation, there is no warranty of non-infringement, no warranty of merchantability,
and no warranty of fitness for a particular purpose. all warranties are expressly disclaimed.
The user assumes the full risk of using this specification. In no event shall any member of
BiDiB-Implement-Group be liable for any actual, direct, indirect, punitive, or consequential
damages arising from such use, even if advised of the possibility of such damages.

BiDiB is intended for use with model railroad control software and may be implemented without license costs at the
PC side as well as on the hardware side. To ensure compatibility, the following license terms & conditions
have to be observed:

Prior to any publication or intended implementation, the BiDiB working group must be informed.
This information will be considered confidential.

The functions described in the following (unless indicated as optional) must be fully implemented and it must
be clearly indicated which features and classes are supported (Example: support of
BiDiB protocol version x.xx, occupancy detection including address and direction).

The used vendor IDs and product IDs must be published, together with a short description of the product,
the supported classes and a link to the description of the product to (manual / data sheet) at bidib.org.
PC programs may only implement a subset (i.e. the functions for firmware update may be omitted in layout
control programs.
However, if a certain function is implemented, this must done in general way and must be conform to the
specification.

Any extension of the protocol must be agreed in advance by the BiDiB working group. Furthermore, the protocol
and reference implementation must be disclosed. This limitation is primarily to ensure compatibility and clarity,
not to block competitors or preventing any new features.

The encoding of data, which are transmitted over BiDiB must be disclosed and freely usable. Exceptions are only
allowed in exceptional cases (i.e. firmware-update). This limitation is primarily to ensure compatibility and
clarity, not to block competitors or preventing any new features.

Each implementation must be linked to the reference document (-> BiDiB.org).
This reference must be made either in the product brochure / data sheet and in the manual (whatever the medium is).

Each publication elsewhere (even partly) requires a written permission and must be linked to the reference
document -> BiDiB.org.

Devices using BiDiB must be marked with the logo. This applies specially to interface connectors, on which BiDiB
is implemented. The logo may be used only with the explicit permission of the BiDiB working group. This limitation
is primarily to ensure compatibility and clarity, not to block competitors or preventing any new features.

Equipment and/or software that implements BiDiB must be provided free of charge in case of an compatibility
check request.

The licensee is obligated to carry out necessary tests regarding the interoperability
of its components (both hardware and software) with the necessary care.
In case of compatibility problems, the corresponding components must be improved
and all necessary information and materials about the troubleshooting results and performed tests
must be provided free of charge.
The licensee agrees to solve the incompatibility within 2 months.
This period may be extended to the next release in case the components is a pure software product.
If the problem is not solved within the above period, the license for this product can be revoked.
In most cases, the interaction between several components is the reason of incompatibility.
The involved parties are obliged to investigate the problem together in the first step
and clarify why these components does not work together.
If an final agreement can't be reached, the issue shall be submitted
to the BiDiB working group in order to decide for further action.

Conscious violations of the license conditions will result in a withdrawal of the license for all products
of the licensee.

1.4. Revision status

This document describes revision 1.27 of BiDiB, updated apr 07, 2017.

This document describes the basic message structure.
Messages that are specifically associated with an application (e.g. switching, detection, driving,...)
will be described in other documents or chapters.

Extension for heterogeneous feedback modules, Individualisation of feature sets for single modules

V0.01

2010-11-03

Initial document

1.5. Design principles

BiDiB is designed as a stateless protocol. There exist several safeguards to deal with message losses.

The most important property to achieve a high error tolerance is the idempotence of the messages.
Generally messages transmit states, not commands or orders.
Their reception is typically acknowledged by the node with a message of equal content.
Thereby the transmission can simply be repeated in case of an error, without causing undesired actions when received multiple times.
This holds in particular for the critical model railroad applications driving,
controlling and feedback.
An identifier is always used in sequential transmissions with multiple messages, so that missing information can be requested again.

Communication errors on the other hand are reported once only and not repeated, which simplifies the implementation.
A loss of an error message can be handled the same way as the absence of the regular answer.
If multiple errors occur together, they can be put in a queue to be retrieved, but they don't have to be repeated either.
If an error persists, it simply is reported again at the next attempt of using the object.
A recovery from an error is generally not reported.
Some objects where fault/error conditions are intrinsic (e.g. boosters and accessories) constitute exceptions hereof,
retention or repetition might be required for them.

Likewise dealt with are spontaneous events that are not mission-critical.
The loss of such messages can only be detected by checking sequence numbers,
but there is no mandated error handling and no way to retransmit the message.
For operation critical events (Occupancy, Accessory) there are acknowledgement mechanisms "Secure-ACK" and "Secure-Switch" as safeguards.

The continuing revolution of BiDiB is deep-seated in the protocol.
Rigorous attention is paid to backward compatibility of new revisions of the specification.
Extensions to the functional range are usually readily possible, as they are typically optional and can be ignored.
In case of backward incompatible changes the protocol version is incremented, which can be retrieved from every node.
This does allow the support of out-dated implementations and even assures the compatibility with unknown functions to some degree.

1.6. Glossary

The following terms for each protocol participant or properties will be used in this document:

BiDiB:

The protocol standard, the way how messages are encoded.

BiDiBus:

One possible physical implementation, specially for wired connections
within a model railway. BiDiBus is based on RS485 and uses RJ45 cables.

Nodes are divided in classes, depending on their basic properties.
There are as example: occupancy detectors, DCC Generators,
Switch outputs, Track panels.

Detector:

A single node within the BiDiB system which is able to detect track occupancy
and recognize BiDi feedback signals.

Feature:

Particular property of a node (e.g. 'detector can recognize loco direction').
Features can be queried and configured individually.

Host:

The control computer, usually a standard PC with appropriate software.

Hub:

A node in one level of the bus system, which is also an interface to the next lower level.

Interface:

The location within a level (structure) in the bus system, which communicates
with the host or parent-node.

Logon:

The attempt of a node to get a logical connection to an interface.
The logon is acknowledged and the host is notified about the change.

Node:

A subscriber inside the BiDiB system (occasionally distributed) hardware.
A subscriber within a level might be also an interface for the next sub-level (Hub-functionality).

Unique-ID, Unique-ID:

The globally biunique node identifier.
It contains the announced classes, the manufacturer code,
and a vendor-specific number derived from the hardware ID.

Node-address:

The automatic assigned number from the interface (byte), under which the node is
(at this level) addressed in this session. (NODE_ADDR)

Magic:

A (meaningless) number, which will be queried by the host during interface startup.
For example the correct baud setting can be recognized. (=system id)

1.7. Principles of node assignment, addressing

An system addressed with BiDiB is organized like folders: The host provides an interface that allows it
to establish a communications link and the interface allows him access to a (flat) array of nodes.
Each of these nodes is usually a component that contains a particular function (for example, a detector
with feedback contacts). But it is also possible that this block itself is an interface too, which holds
a further structure (like subfolders). This achieves high flexibility in wiring possibilities and also in
the heterogeneity of the connected detectors.

BiDiB is able to assign addresses automatically to nodes. Each node has an unique manufacturer-programmed number that is part of the Unique-ID.
During startup of the feedback system, the interface searches existing nodes within its structure and assign a
local address to each node. This address has the length of one byte. The interface builds a assignment table
with all available nodes, their Unique-ID and their local address. Nodes itself can be also interfaces which
gives a threaded structure. The maximum address length within the thread is 4 bytes.

This assignment table is different on each startup and will be automatically extended if a new node is added to
the bus. In this case, the interface transmits a message to the host.

Internal to the host, the nodes are also stored with their unique ID, besides the program-specific things like screen
position and node name. The host will retrieve the mapping from the interface during startup and gets
the valid local addresses for this session that belongs to the unique IDs.
The mapping between the object on the user interface and the model hardware is done through the node identity and is
therefore independent from the current address assignment.

To facilitate the replacement of nodes, it should be minded in host programs
that the Uniqued-ID assigned to a node object can easily be exchanged for another one.
Because the unique ID of a device cannot be set by the user,
they cannot program the replacement node to the "old address"
to continue using the mapping configured in the program;
instead it is necessary to make the change within the host software.
The swapping of the unique ID is needed for instance
when replacing a defective device by a identically constructed one,
when upgrading hardware to a superior product with more outputs,
or when deploying a different product firmware for more functionality.

How does the first time node assignment is working?

If the host reads the mapping table and he found one ore more new node(s) which are not listed in the layout,
he may show this newly available nodes in a separate pool of 'new detectors/command stations/boosters etc.'.
For example, you can trigger a flashing of the occupancy detection (or Identify) by pressing the identify button
on the target node and then assign the correct section of track in the PC.

What happens if a node with occupancy functionality fails or has been replaced?

The PC can display an error message if he reads the mapping table and miss a unique ID for an already
configured occupancy detector. It might be also possible that a new indicator appears in parallel. In this case,
the program could ask whether this new detector replaces the old one right away.

2. Physical implementation

BiDiB protocol is suitable for different transmission media such as
serial connection, USB, RS485,
Ethernet or wireless (with an adapted framing for each media). For specifications, refer to the respective documents.

The respective physical layer ensures the correct framing and bytewise, transparent transport of BiDiB message packets.
The transfer layer must secure the transport of the data with CRC. The transfer layer must at least be able to transport
messages with a size of 64 bytes.

CRC means the CRC8-Byte; On the transmitter side, the polynom x8 + x5 + x4 + 1
will be generated over the message, starting at the first byte from the message, Init=0, none inverted.
On receiver side, the CRC with the same polynom will be generated over the whole message including CRC. The result
must be 0.

A message contains the length, an address specification, a (consecutive) message number, a type specification
and optional parameters. These fields are described below:

MSG_LENGTH contains one byte and describes the message size in bytes.
Here, the length is specified from the first byte of MSG_ADDR until the end of the DATA field.
A message which contains only address 0, MSG_NUM and MSG_TYPE has therefore the length 3.
Value range 0…127, MSB is reserved.Note: If future enlargement requires an increasing of the messages size, the reserved MSB can be used
for this extension. For example – according to the following rule:
If the first byte in the MSG_LENGTH field is less than 128, then the MSG_LENGTH field is only one byte.
If the MSB is set in the first byte of the MSG_LENGTH field, the length will be calculated as follows:
(First byte & 0x7F) * 128 + (second byte & 0x7F); The MSB may not be set in the 2nd byte.

MSG_ADDR describes the assigned destination or source address of this message.
MSG_ADDR consists of 1 to 4 bytes, each byte defines the local address (NODE_ADDR) on a level.
The last byte of MSG_ADDR is always 0.
If the MSG_ADDR equals 0 (so the MSG_ADDR_STACK is empty), the message is addressed directly to the interface or node.
If the MSG_ADDR doesn't equal 0, the one node behind the interface will be addressed
whose local address corresponds to the first NODE_ADDR.
In this case, the interface forwards the message accordingly, at which the MSG_ADDR_STACK will be reduced by one byte.
Conversely, for messages that are received from a lower node the interface prepends the local node address.

MSG_NUM denotes a continuous message index, which is incremented by one with each message.
The 0 is skipped (1,2,...,255,1,2,...). The receiver is able to check the MG_NUM and can determine whether
if a message has been lost. In the event of lost messages, it is up to the receiver how he react.
Restart can be made with the protocol commands, a retransmit of single messages is not provided.
Usually, the receiver will request all occupancy states again and if Secure ACK is enabled, the repeat
will be made automatically.
When MSG_NUM is transmitted with value 0, then the transmitter skip the verification of the sequence,
an appropriate error message is not allowed.
The 0 resets also the sequence in the receiver, after 0 it will start with the continuous message
index at 1.
In broadcast messages, the MSG_NUM is completely ignored, it is preset to 0.

MSG_TYPE: A byte which encodes the message type and defines in principle what kind of
message is available here.
For faster decoding, the message types are summarized in groups:

0x00

Offset for downlink messages

0x80

Offset for uplink messages

0x00

Offset for system messages

0x10

Offset for feature and user messages

0x20

Offset for feedback detector messages

0x30

Offset for booster messages

0x38

Offset for accessory, switch and macro messages

0x60

Offset for DCC generator messages

0x70

Offset for local messages

The detailed message coding can be found in the bidib_messages.h file, which is available for
all licensees.

DATA: depending on MSG_TYPE, other parameters of the message will follow.
There are all bytes 0…255 allowed for DATA.

The data securing against transmission errors is the responsibility of the respective transport medium
(e.g. CRC in case of serial transmission).
The protocol itself doesn't provide message retransmit, critical messages are secured at a higher level
(e.g. with Secure-ACK or corresponding response of the node).

The message length (MSG_LENGTH) does not only delimit messages where multiple are sequenced back-to-back,
but also specifies how many DATA bytes are actually available as message parameters.
This might indeed be more or less bytes than the message type suggests.
Thereby optional parameters can be accomplished that are set to default values if not existent.
If less parameters are sent than necessary, the node responds with an error message.
If more parameters are sent than expected, the surplus bytes are ignored; this assures forward compatibility.

A message may have a length up to 64 bytes (MSG_LENGTH=63), all transport layers need to support these.
Longer downstream messages can, as long as supported by the respective transport layer and node,
be allowed per node via the MSG_PKT_CAPACITY message.
To send a long message to a node, all interfaces on the transport path have to support this as well.
Longer upstream messages shall be limited in accordance with the lengths used in the downstream.

3.2. Routing

BiDiB allows to wire subnets and connect them in star topology via hubs (similar to USB).
This provides scalability for large model layouts.
For smaller layouts, probably only one network will be build and therefore the information in this
chapter can be skipped for now.

Local address assignment in sub-networks is described in a further document
For the protocol description, only the results of this process are relevant.

A maximum of 4 cascaded networks is allowed. This limitation is intended to facilitate host implementations,
thereby the respective current local address of a endnode is limited to 32 bits.

The following rules apply for the routing of messages:

Downstream: A node that receives a MESSAGE will check the destination address of this message: it contains its local
NODE_ADDR in the current structure, possibly followed by further sub addresses.
If the next byte is not 0 (belonging to MSG_ADDR_STACK), the node transmits the MESSAGE to the next subnode,
but the first byte (his own NODE_ADDR) will be removed.
If the next byte is equal to 0 and the MSG_ADDR_STACK is empty, the node itself is addressed.
For broadcast messages, the MSG_ADDR_STACK is always empty.

Upstream: A node which receives a MESSAGE from a sub-node, adds the local address (NODE_ADDR) of the subnode to the
message and transmits them to its interface.

To enable the parent node, respectively the host, to detect this structure, appropriate commands are existing:

MSG_NODETAB_GETALL, MSG_NODETAB_GETNEXT: Associated sub-nodes from a node will be queried with this command.
The node answer this query with a MSG_NODETAB_COUNT (indicates the size of the table), the individual table entries
are requested with a MSG_NODETAB_GETNEXT, this will be answered with a single MSG_NODETAB message in each case.
If the node has no sub-nodes at all, the length of the node table is 1 and the only included entry is the node itself.
If a power-up event has been send but the bus arbitration in the sub-node structure is not yet complete, the query
will be answered with MSG_NODETAB_COUNT = 0. The node table request should be answered only after the sub-node structure
has become stable. Frequent messages about state changes at the start can be avoided.

If there is a change in the assignment table during operation (e.g. connect or disconnect), the interface sends
a MSG_NODE_LOST or MSG_NODE_NEW message anyway. This message must be acknowledged by MSG_NODE_CHANGED_ACK (or a complete node table
query). Both messages MSG_NODE_LOST, MSG_NODE_NEW and also MSG_NODE_CHANGED_ACK have the (continuous) version
number of the table as a parameter. This ensures that all changes at the bus structure will be recognized.

Changes in the node table should be avoided by the interface (e.g. transmit no logon request).
If there is a change in the node table while being queried and transmitted with MSG_NODETAB_GETNEXT, the interface
must start over again with the transmission of a new MSG_NODETAB_COUNT (but with an incremented version number).

3.3. Unique-ID

Each node has a distinct identifer, this number is called Unique-ID.
The Unique-ID contains 7 bytes:

Structure of Unique-ID

1. Byte

ClassID

This is a bit field indicating the class membership of this node.
A node may also belong to several classes at once.

The classes serve as a quick reference for the host
about which functionalities can be found on this specific node.
To rapidly locate a feature-based subfunctionality it is enough
to only query the nodes which have set the corresponding class bit.

If a node has implemented commands of a particular class,
the appropriate class bit must be set as well.
Conversely, it must know the commands of the announced classes and answer them correctly.
Even if in the current configuration no objects are available,
it should register the class and yield 0 for the count.

These 4 bytes (= 32 bits) are split into a product identifier (lower 'p' bits) and
a serial number (upper 's' bits). This allows for an easier identification of nodes by
analysis tools and host programs. The coding in BiDiB is always little-endian (low-byte first),
the product identifier starts at bit 0, first byte. It is up to the vendor, how many bits
he uses for product and for serial number. However, a default of 16 bit / 16 bit is recommended.
The feature FEATURE_RELEVANT_PID_BITS defines, how many bits are used for product ('p').

If the feature FEATURE_RELEVANT_PID_BITS doesn't exist, the default of 16 bits / 16 bits is used.

The uniqueness of the product ID (and with it, the Unique-ID) is guaranteed
by every vendor through the hardcoded serial numbers.
Neither the unique ID nor the product ID constitute a fixed hardware identifier however,
one and the same device can have several firmwares installed
which may differ in their classes or even product types.
Only the serial number remains the same normally,
but on its own that is not necessarily unique.

Representation of the unique-ID at host programs / stickers / tools:

A unified representation of the unique-ID is recommended, so users can easily assign modules.
Vendor-ID should be detached from the unique-ID and the 4. until 7.byte (in this order) should
be displayed as a HEX value.

Examples:

VID 0D PID 0278456B

Vendor: Public Domain & Do-It-Yourself Decoders, Unique ID 0x0278456B

Additionally, there might be a user string stored in the node (nick name, see MSG_STRING). This name can be used as alternative
for the presentation of the node.

3.4. Typical protocol startup

Connecting to a BiDiB system can be done according to the following list:

Connection to the first node (usually the interface).
For example, with a serial interface: The host sends two MAGIC characters,
followed by a MSG_SYS_GET_MAGIC message. This will be made with the possible
baud rates and determine whether a error free response is coming from the
interface. A 300ms timeout is enough at this point.

Stopping the BiDiB system.
By sending a MSG_SYS_DISABLE, the spontaneous transmission of messages is suppressed.
This is primarily used to avoid interruption through spontaneous messages in the
following read-out phase.
Alternatively, the system can also execute a complete restart of the node including
new assignment through MSG_SYS_RESET.

Now we read the properties of the first node.
Important here is the answers to MSG_GET_SYS_MAGIC, MSG_SYS_GET_P_VERSION and the
class bits from the Unique-ID, that indicate the properties of the node and which
messages it does support.
If the node is an interface, the node list must be queried (MSG_NODETAB_GETALL).
Furthermore, you can query all features and other information of this node (MSG_FEATURE_GETALL).
Please note that the simultaneous queries for a single node are limited, the expected response
may not exceed the size of 48 bytes.
If a node supports configuration fingerprinting,
you can quickly check with the answer to MSG_SYS_GET_UNIQUE_ID
whether the settings still match the already known ones.

Additional nodes.
If additional nodes have been reported, step 3 must be repeated for all nodes that have
been obtained in the node list. In case there is again a node with an interface, this
interface will be also queried for the node list.
It is possible to handle multiple nodes in parallel. For example, transmitting a MSG_FEATURE_GETNEXT
for each MSG_FEATURE immediately. A 500ms timeout is enough at this point

Release.
If all data has been read, then the system can be released for spontaneous messages
with MSG_SYS_ENABLE.

3.5. Classes

Nodes declare different classes via their Unique ID, e.g. occupancy detection, hub or booster functionality. To what extent a class is implemented is announced through feature settings.

For each class there are mandatory messages that have to be implemented, as well as optional messages that are only available if a feature declares them to be.

Class Interface:

An Interface (Hub) transports messages from and to further nodes on a sub level.

Mandatory:

All system messages

All messages for node management (NEW, LOST, etc.)

Overload safety: An interface must not lose messages, appropriate buffers need to be provided.

A booster amplifies the DCC signal for decoders (but does not generate it).

Mandatory:

All system messages

FEATURE_BST_INHIBIT_AUTOSTART

Messages for boosters

Optional:

Diagnostics, FEATURE_BST_CURMEAS_INTERVAL and MSG_BOOST_DIAGNOSTIC

BiDi-detection, features and messages

User configuration

Firmware update

Class Peripherals:

The node provides switching and animation functionality for peripheral equipment and lights.

Mandatory:

All system messages

FEATURE_CTRL_*_COUNT or FEATURE_CTRL_PORT_FLAT_MODEL

Messages for port commands and configuration

Optional:

FEATURE_CTRL_PORT_QUERY_AVAILABLE and messages for port queries

FEATURE_CTRL_INPUT_NOTIFY and spontaneous input events

FEATURE_CTRL_MAC_LEVEL and messages for macro controls

User configuration

Firmware update

3.6. System time

A BiDiB system has a global system time available to achieve a higher accuracy in time-based measurements.
Various messages may contain a timestamp for the respective event,
this allows for better results in vehicle tracking and calibration.
The uncertainty in the latency of the data transmission is eliminated.

The BiDiB system time has a resolution of 1 millisecond and is represented by a 16-bit integer with cyclic overflow
An error margin of ±5 ms between the clocks of all nodes in a system is aspired.

To facilitate a time measurement across different nodes, the clocks of the nodes have to be synced.
This synchronisation happens on every bus level depending on the transport layer and is carried out by the respective interface.
Whether an interface/hub does or not support it must be stated in the product description.

The synchronisation typically happens through a periodically emitted broadcast message with a timestamp.
On the receipt of this message, the local node clock is set to the system time.
Delays on the communication path or internally at the transmission and reception need to be considered
and have to be corrected through according offsets,
the arising inaccuracy should be ±1ms per bus level at most.
With this approach a (small) step in the time series at the point of adjustment may occur,
a host will need to take such into account.

If the synchronisation is absent, a node must refrain from the use of timestamps
as soon as the resulting uncertainty becomes larger than ±3 ms.
Example: if a processor clock rate has a divergence of up to 10ppm (0,01 ‰),
then it should not use its local clock from about 100 s after the last synchronisation.

An interface should wait for synchronisation from the upper level before supplying the system time to its subnodes.
If a synchronisation message fails to appear until the SYS_ENABLE (or a reasonable timeout),
the interface may use its local clock time and synchronise the bus structure on its own.

4. Messages

The messages, which are used on the BiDiB system, can be divided into the following categories: