Mr.Raindrop

The MQTT protocol has actually been in use for a long time, and recently I had the opportunity to familiarize myself with the entire protocol system. The main reference was the introduction document of the MQTT 3.1.1 protocol in both Chinese and English.

mqtt-v3.1.1-errata01-os-complete.pdf

Chapter 1 - Introduction to MQTT · Chinese Version of MQTT Protocol

Application Message The MQTT protocol transmits application data over the network. When application messages are transmitted via MQTT, they have associated Quality of Service (QoS) and topics.

Client: Publishes messages and subscribes to requests.

Server: Receives and publishes messages, and handles subscriptions.

Subscriptions include a topic filter and a maximum Quality of Service (QoS) level.

Control Message Format#

The MQTT protocol communicates by exchanging predefined MQTT control messages.

Control messages mainly include: fixed header, variable header (partially included), and payload (partially included).

Control Message Types#

Fixed Header#

The fixed header is two bytes.

• The first byte: The high four bits are the control message type, while the low four bits are the corresponding control message type's flags
• The second byte (remaining length): Indicates the byte count of the remaining part of the current message, including the variable header and payload data. The remaining length does not include the bytes used for encoding the remaining length field itself.

Variable Header#

Located between the fixed header and payload. Many control messages' variable header parts includea two-byte message identifier field.

Payload#

Located at the end of the message.

MQTT Operation Flow#

Client Connection#

The client sends a connect message to the server

Including client identifier (must), username, password, protocol level, and will information, etc.

The server sends a CONNACK to the client to confirm the connection request

Subscribe to Topic#

The client sends a SUBSCRIBE message to create one or more subscriptions (supports wildcards). The SUBSCRIBE message also specifies the maximum QoS level for each subscription, and the server sends application messages to the client based on this.

SUBACK is used for subscription acknowledgment.<font style="color:rgb(0, 0, 0);">SUBACK</font> message will return a granted QoS level for each topic filter in the <font style="color:rgb(0, 0, 0);">SUBSCRIBE</font> message** (this level may be equal to or less than the maximum QoS requested by the client, but will not exceed it).

PUBLISH Send Message#

The client uses the PUBLISH message to send application messages to the server, with the purpose of distributing to other subscribed matching clients.

The server uses the PUBLISH message to send application messages to each subscribed matching client.

The variable header includes the topic name and message identifier, while the payload contains the message to be published.

• When QoS is 0, there is no acknowledgment process.
• When QoS is 1, acknowledgment is required through PUBACK.
• When QoS is 2, it first requires acknowledgment of receipt through PUBREC, then uses PUBREL for release acknowledgment, and finally uses PUBCOMP for completion acknowledgment.

Unsubscribe#

Use UNSUBSCRIBE to unsubscribe.

Use UNSUBACK to confirm receipt of the UNSUBSCRIBE message.

Disconnect#

The client sends a DISCONNECT message to disconnect from the server.

Control Messages#

CONNECT – Connect to Server#

• The first message sent by the client to the server after establishing a network connection must** be the CONNECT message
• On a network connection, the client can only send the CONNECT message once.
• The payload contains one or more encoded fields, including the client's unique identifier, Will topic, Will message, username, and password.

Fixed Header#

The first byte: message type 0001, with the low four bits as reserved bits.

The remaining length equals the length of the variable header (10 bytes) plus the length of the payload.

Variable Header#

The variable header of the CONNECT message contains four fields in the following order: Protocol Name, Protocol Level, Connect Flags, and Keep Alive.

Protocol Name: Indicates the encoding of MQTT.#

Protocol Level: Indicates the protocol version.#

Connect Flags#

• Clean Session is

0: The server must restore communication with the client based on the current session (identified by the client identifier).

1: After disconnection, the old session will be discarded.

To ensure that messages during the disconnection period are not lost, QoS 1 or QoS 2 levels must be used, while setting the clean session flag to 0.

• Will Flag: Indicates whether the server needs to publish the will message.
• Will QoS: QoS level of the will message.
• Will Retain: Indicates whether the will message is published as a retained message.
• Username Flag: Indicates whether the payload contains a username.
• Password Flag: Indicates whether the payload contains a password.

Keep Alive#

Keep Alive is a time interval in seconds.

It indicates the maximum allowed idle time interval between the moment the client completes transmitting a control message and the moment it sends the next message. During this time, if no other control messages can be sent, the client must send a PINGREQ message.

If the keep alive value is non-zero and the server does not receive a control message from the client within one and a half times the keep alive time, it must disconnect the client's network connection, considering the network connection to be disconnected.

After the client sends the PINGREQ message, if it still does not receive the PINGRESP message within a reasonable time, it should close the network connection to the server.

A keep alive value of zero indicates the disabling of the keep alive feature.

Payload#

The payload of the CONNECT message contains one or more length-prefixed fields, and the flags in the variable header determine whether these fields are included. If included, they must appear in this order: client identifier, will topic, will message, username, password.

Client Identifier#

The server uses the client identifier (ClientId) to identify the client. Each client connecting to the server has a unique client identifier (ClientId). Both the client and server must use the ClientId to identify the state related to the MQTT session between them.

The client identifier (ClientId) must exist and must be the first field of the CONNECT message payload.**

Will Topic#

If the Will flag is set to 1, the next field in the payload is the Will Topic. After that is the Will message, which may include the username and password depending on the flags.

Response#

The server can support multiple protocols on the same TCP port or other network endpoints.

The server must send a CONNACK message with a return code of zero as a confirmation response to the CONNECT message. However, the client can send control messages before receiving the CONNACK.

CONNACK – Confirm Connection Request#

The server sends a CONNACK message in response to the CONNECT message received from the client. The first message sent by the server to the client must be CONNACK.

Fixed Header#

Variable Header#

Connection Acknowledgment Flags#

The 0 (SP) bit is the current session (Session Present) flag. Other (1-7) bits should be 0.

If the server receives a connection with the clean session (CleanSession) flag set to 1, in addition to setting the return code in the CONNACK message to 0, it must also set the current session flag (Session Present) in the CONNACK message to 0.

When the server receives a connection request with CleanSession set to 0, it needs to determine the current session flag in the CONNACK response based on whether it has saved the session information for that client (identified by ClientId). If it has previously saved the session information for this client, then the current session flag in the CONNACK should be set to 1; if not, it should be set to 0. In either case, the return code in the CONNACK must be ensured to be 0.

Connection Return Code#

An unsigned value, 0 indicates a normal connection; non-zero values indicate that the connection was refused.

No Payload#

Publish - Publish Message#

Fixed Header#

Resend Flag DUP: 0 First request sent, 1 indicates a retransmission.#

The sent (outbound) PUBLISH message and the received (inbound) PUBLISH message's DUP flag are independently set, and its value must be determined separately based on whether the sent (outbound) PUBLISH message is a retransmission.

Quality of Service Level QOS#

Retain Flag RETAIN#

If the PUBLISH message sent by the client to the server has the retain (RETAIN) flag set to 1, the server must store this application message and its Quality of Service (QoS) level so that it can be distributed to future subscribers matching the topic name.

When a new subscription is established, if there is a recently retained message for each matching topic name, it must be sent to this subscriber.

If the server receives a retained (RETAIN) flag set to 1 QoS 0 message, it must discard any previously retained messages for that topic. It should treat this new QoS 0 message as the new retained message for that topic, but it can choose to discard it at any time — if this happens, that topic will have no retained message.

When the server sends a PUBLISH message to the client, if the message is sent as a result of a new subscription by the client, it must set the message's retain flag to 1.

Remaining Length Field#

The length of the variable header plus the length of the payload.

Variable Header#

Contains the topic name and message identifier.

Topic Name#

The first field of the variable header.

The topic name cannot contain wildcards.

Message Identifier#

Exists when QoS is 1 or 2.

Payload#

The payload contains the application message to be published. The content and format of the data are application-specific.

Response#

The receiver's QoS level should match that of the publisher.

Action#

The client uses the PUBLISH message to send application messages to the server, with the purpose of distributing to other subscribed matching clients.

The server uses the PUBLISH message to send application messages to each subscribed matching client.

When the client requests to subscribe using a topic filter with wildcards, the client's subscriptions may duplicate, so the published message may match multiple filters. In this case, the server must distribute the message to all clients matching the highest QoS level.

The server can then distribute copies of the message to each matching subscriber according to the QoS level of the subscription.

PUBACK – Publish Acknowledgment#

Response to QoS 1 level PUBLISH message.

Fixed Header#

MQTT message control type is 4.

The remaining length field is 2.

Variable Header#

Contains the message identifier of the PUBLISH message waiting for acknowledgment.

No Payload#

PUBREC – Publish Received (QoS 2, First Step)#

Response to QoS level 2 PUBLISH message. It is the second message in the QoS 2 level protocol exchange.

Fixed Header
MQTT message control type is 5
The remaining length field is 2

Variable Header#

Contains the message identifier of the PUBLISH message waiting for acknowledgment.

No Payload

PUBREL – Publish Release (QoS 2, Second Step) #

Response to the PUBREC message. It is the third message in the QoS 2 level protocol exchange.

Fixed Header#

MQTT message control type is 5

The remaining length field is 2

Variable Header#

The same message identifier as the PUBREC message waiting for acknowledgment.

No Payload

PUBCOMP – Publish Complete (QoS 2, Third Step) #

Response to the PUBREL message. It is the fourth and final message in the QoS 2 level protocol exchange.

Fixed Header
MQTT message control type is 7
The remaining length field is 2
Variable Header

The same message identifier as the PUBREC message waiting for acknowledgment.
No Payload

SUBSCRIBE - Subscribe to Topic #

The client sends a SUBSCRIBE message to the server to create one or more subscriptions. The SUBSCRIBE message also specifies the maximum QoS level for each subscription, and the server sends application messages to the client based on this.

Fixed Header
MQTT message control type is 5
The remaining length field is the length of variable header + payload

Variable Header#

The variable header contains the client identifier.

Payload#

Topic filter and QoS level field combination

Indicates the topics the client wants to subscribe to. The server should support topic filters that include wildcards (as defined in section 4.7.1). If the server chooses not to support topic filters that include wildcards, it must reject any subscription requests that include wildcard filters.

Each filter is followed by a byte, known as the requested QoS. It indicates the maximum QoS level allowed for the server to send application messages to the client. The payload must contain at least one pair of topic filter and QoS level field combination.

A SUBSCRIBE message with no payload is a violation of the protocol.

Response#

When the server receives a SUBSCRIBE message sent by the client, it must respond with a SUBACK message, which must have the same message identifier as the SUBSCRIBE message waiting for acknowledgment.

The topic filters of the new subscription may be the same as those previously subscribed, but their maximum QoS values may differ. Any existing retained messages matching this topic filter must be resent, but the publishing process cannot be interrupted.

If the server receives a SUBSCRIBE message containing multiple topic filters, it must handle it as if it received a series of multiple SUBSCRIBE messages, except that it needs to merge their responses into a single SUBACK message. The SUBACK response can be merged.

The SUBACK message must include a return code for each pair of topic filter and QoS level.

The server can grant a lower QoS level than requested by the subscriber. The QoS of the payload of the message sent in response to the subscription must be the minimum of the QoS of the original published message and the QoS granted by the server.

Determining the possible maximum QoS level during message distribution is the responsibility of the publisher, while the subscriber can request the server to lower the QoS to a level more suitable for it.

SUBACK – Subscription Acknowledgment#

Used to confirm that it has received and is processing the SUBSCRIBE message. It contains a list of return codes that specify the maximum QoS level granted for each subscription in the SUBSCRIBE request.

Fixed Header

MQTT message control type is 5

The remaining length field is the length of variable header + payload

Variable Header#

Contains the message identifier of the SUBSCRIBE message waiting for acknowledgment.

Payload#

Contains a list of return codes. Each return code corresponds to a topic filter in the SUBSCRIBE message waiting for acknowledgment. The order of return codes must match the order of topic filters in the SUBSCRIBE message.

Allowed return code values:

0x00 - Maximum QoS 0

0x01 - Success – Maximum QoS 1

0x02 - Success – Maximum QoS 2

0x80 - Failure

UNSUBSCRIBE – Unsubscribe#

Used to unsubscribe from topics.

Fixed Header#

Control message type 10

Reserved bits 0, 0, 1, 0

Remaining length field is the length of variable header + payload

Variable Header#

Payload#

Contains the topics the client wants to unsubscribe from.

Response#

If the server deletes a subscription:

• Stops distributing any new messages to this client
• Completes the distribution of any QoS 1 and QoS 2 messages that have already started sending to the client
• Continues to send any existing cached messages prepared for distribution to the client.

The server must send an UNSUBACK message in response to the client's UNSUBSCRIBE request.

UNSUBACK – Unsubscribe Acknowledgment#

Used to confirm receipt of the UNSUBSCRIBE message.

Fixed Header#

Control message type 11

Remaining length 2

Variable Header#

Payload#

No payload

PINGREQ – Heartbeat Request#

• Notifies the server that the client is still alive.
• Requests the server to send a response confirming it is still alive.
• Uses the network to confirm that the network connection is not broken.

Fixed Header#

Control message type 12

Remaining length 0

No variable header, no payload

Response#

The server must send a PINGRESP message in response to the client's PINGREQ message.

PINGRESP – Heartbeat Response#

Sends a PINGRESP message in response to the client's PINGREQ message. Indicates that the server is still alive.

Fixed Header#

Control message type 13

Remaining length 0
No variable header, no payload

DISCONNECT – Disconnect #

Control message type 14

Reserved bits 0

Remaining length 0 No variable header, no payload

Response #

After the client sends the DISCONNECT message:

• Must close the network connection
• Cannot send any control messages over that network connection again

When the server receives the DISCONNECT message:

• Must discard any unprocessed will messages associated with the current connection
• Should close the network connection if the client has not done so already.

Operational Behavior#

The transport layer protocol used by MQTT 3.1 is the TCP/IP protocol defined in [RFC793]. It also supports TLS and WebSocket protocols.

Quality of Service Levels and Protocol Flow#

QoS 0: At most once#

Message delivery depends on the capabilities of the underlying network. The receiver does not send a response, and the sender does not retry.

QoS 1: At least once#

The variable header of the QoS 1 PUBLISH message contains a message identifier, which requires PUBACK message acknowledgment.

Sender:

• Must assign an unused message identifier each time a new application message is sent.

• The sent PUBLISH message must contain a message identifier and have QoS equal to 1, with DUP equal to 0.

• This PUBLISH message must be considered unacknowledged until the corresponding PUBACK message is received from the receiver.

Receiver:

• The responding PUBACK message must contain a message identifier that comes from the received PUBLISH message that has accepted ownership.

• After sending the PUBACK message, the receiver must treat any inbound PUBLISH messages with the same message identifier as a new message and ignore its DUP flag value.

QoS 2: Exactly once#

Message loss and duplication are unacceptable.

Sender:

• Must assign an unused message identifier for the new application message to be sent.
• The sent PUBLISH message must contain a message identifier and have QoS equal to 2, with DUP equal to 0.
• This PUBLISH message must be considered unacknowledged until the corresponding PUBREC message is received from the receiver.
• After receiving the PUBREC message, it must send a PUBREL message. The PUBREL message must contain the same message identifier as the original PUBLISH message.
• This PUBREL message must be considered unacknowledged until the corresponding PUBCOMP message is received from the receiver.
• Once the corresponding PUBREL message is sent, the PUBLISH message cannot be retransmitted.

Responder:

• The responding PUBREC message must contain a message identifier that comes from the received PUBLISH message that has accepted ownership.
• Before receiving the corresponding PUBREL message, the receiver must send a PUBREC message acknowledging any subsequent PUBLISH messages with the same identifier. In this case, it cannot redistribute messages to any subsequent receivers.
• The PUBCOMP message responding to the PUBREL message must contain the same identifier as the PUBREL message.
• After sending the PUBCOMP message, the receiver must treat any subsequent PUBLISH messages with the same message identifier as a new publication.

Message Delivery Retry#

Clients and servers must retransmit any unacknowledged PUBLISH messages (if QoS>0) and PUBREL messages using the original message identifier. This is the only situation that requires clients or servers to retransmit messages.

Topic Names and Topic Filters#

Topic Wildcards#

The slash (‘/’ U+002F) is used to separate each level of the topic, providing a hierarchical structure for topic names.

Multi-level Wildcard#

The number sign (‘#’ U+0023) is a wildcard used to match any level in the topic.

Single-level Wildcard#