MQTT Forward Instruction

* Firmware Version > LG02_LG08-5.3.1580178039 [[Firmware Download>>url:http://www.dragino.com/downloads/index.php?dir=LoRa_Gateway/LPS8/Firmware/Release/]]

16

* LG01N, OLG01N ((% style="color:red" %)**Warning**(%%): LG01-P LG01-S use another instruction: [[MQTT for LG01-P/LG01S>>url:https://wiki.dragino.com/index.php/Through_MQTT_to_upload_data]])

17

* LG02, OLG02

* LG308, DLOS8

* LPS8

* LIG16

* MS14 series if installed with the same firmware. (in this case, the MQTT forward will work , but no LoRa support)

22

23

= 2. Firmware Change Log for MQTT feature =

24

25

(((

26

This instruction is wrote start from LG02_LG08-5.3.1580178039. Below is related change log since this version of firmware.

27

)))

28

29

* LG02_LG08-5.3.1580178039

30

** Initiate version

31

32

= 3. MQTT forward operating principle =

33

34

== 3.1 Network Structure ==

35

36

Below shows the network structure for MQTT forwarding.

37

38

* For Uplink: The sensor sends data to LoRa Gateway via LoRa wireless, The gateway will process these data and forward to remote MQTT Broker via Internet.

39

* For Downlink: The gateway subscribe a topic in the MQTT broker, when there is update on the topic, the gateway will know and broadcast the data to Local LoRa network,

40

41

[[image:image-20220527133547-1.png]]

42

43

General MQTT structure

44

45

46

== 3.2 How sensor data is forwarded ==

47

48

In this MQTT forward feature, the key point is how the gateway process the sensor data.

49

50

51

=== 3.2.1 Upstream ===

52

53

Assume there are two sensor nodes, their ID are Node1 ID: 6734 , Node2 ID: 7456. In the remote MQTT broker there are two topics: Topic1: /channel/765800, Topic2: /channel/367860. We can set up in the gateway to map Node1 to Topic1 and Node2 to Topic2. So when there is a sensor data from Node1, the gateway will forward the data to Topic1, when there is sensor data from Node2, the gateway will forward to Topic2.

54

55

(((

56

The data flow works as below diagram.

57

)))

58

59

[[image:image-20220527134000-2.png]]

(((

Upstream path

)))

(((

Note: The sensor data can base or LoRa or other method, as long as there are data on the file /var/iot/channels. /span>

)))

(((

)))

=== 3.2.2 Downstream ===

74

75

The gateway subscribes to a topic of the remote MQTT broker topic. When there is some one publish a value on this topic. The gateway will get it and broadcast to local LoRa Network.

76

77

(((

78

Below are the data flow for downstream.

79

)))

80

81

[[image:image-20220527134038-3.png]]

Downstream path

== 3.3 Macro Definition ==

87

88

The MQTT publish command use Macro settings to generate flexible upstream payload for MQTT publish.

89

90

(((

91

Currently the (% style="color:#4f81bd" %)**-t (topic)**(%%) and (% style="color:#4f81bd" %)**-m (message)**(%%) support Macros.

)))

=== 3.3.1 -t topic macro ===

97

98

* CHANNEL: Remote Channel ID

99

* CLIENTID: Client ID , Same as -i

100

* WRITE_API: Remote Channel Write API

101

* USERNAME: User ID (-u)

102

* HOSTNAME: Device Hostname

103

104

=== 3.3.2 -m message macro ===

105

106

* HOSTNAME: Device Hostname

107

* CHANNEL: Remote Channel ID

108

* DATA: Sensor Data without time stamp and rssi

109

* META: Completely sensor data with time stamp and rssi

110

* JSON: Convert META to json format.

111

112

=== 3.3.3 Example for Macro ===

113

114

[[image:image-20220527134251-4.png]]

115

116

MQTT Publish configure

117

118

Above screen shots shows below format:

119

120

* -t: CLIENTID/CHANNEL/data

121

* -m: DATA

122

123

When there is a LoRa sensor arrive. it will be store at the /var/iot/channels as below:

124

125

[[image:image-20220527134332-5.png]]

Sensor Data

(((

According to above macro. Gateway will publish (% style="color:#4f81bd" %)**field1=22.0&field2=49.0**(%%) to topic: (% style="color:#4f81bd" %)**dragino-1b7060/78901/data**(%%), where 78901 is the remote channel for this node ID.

)))

== 3.4 Modify the MQTT to support more options ==

135

136

The MQTT Client Utility used in Dragino is **mosquitto_pub** and **mosquitto_sub**. User can add more options to the mqtt commands. User can check the valid options by command mosquitto_pub ~-~-help. as below:

(% class="box" %)

(((

root@dragino-1ec39c:~~# mosquitto_pub ~-~-help

141

mosquitto_pub is a simple mqtt client that will publish a message on a single topic and exit.

142

mosquitto_pub version 1.6.4 running on libmosquitto 1.6.4.

143

\\Usage: mosquitto_pub {[-h host] [-p port] [-u username] [-P password] -t topic | -L URL}

144

{-f file | -l | -n | -m message}

145

[-c] [-k keepalive] [-q qos] [-r] [~-~-repeat N] [~-~-repeat-delay time]

146

[-A bind_address]

147

[-i id] [-I id_prefix]

148

[-d] [~-~-quiet]

149

[-M max_inflight]

150

[-u username [-P password]]

151

[~-~-will-topic [~-~-will-payload payload] [~-~-will-qos qos] [~-~-will-retain]]

152

[{~-~-cafile file | ~-~-capath dir} [~-~-cert file] [~-~-key file]

153

[~-~-ciphers ciphers] [~-~-insecure]

154

[~-~-tls-alpn protocol]

155

[~-~-tls-engine engine] [~-~-keyform keyform] [~-~-tls-engine-kpass-sha1]]

156

[~-~-psk hex-key ~-~-psk-identity identity [~-~-ciphers ciphers]]

157

[~-~-proxy socks-url]

158

[~-~-property command identifier value]

159

[-D command identifier value]

160

mosquitto_pub ~-~-help

161

\\-A : bind the outgoing socket to this host/ip address. Use to control which interface

162

the client communicates over.

163

-d : enable debug messages.

164

-D : Define MQTT v5 properties. See the documentation for more details.

165

-f : send the contents of a file as the message.

166

-h : mqtt host to connect to. Defaults to localhost.

167

-i : id to use for this client. Defaults to mosquitto_pub_ appended with the process id.

168

-I : define the client id as id_prefix appended with the process id. Useful for when the

169

broker is using the clientid_prefixes option.

170

-k : keep alive in seconds for this client. Defaults to 60.

171

-L : specify user, password, hostname, port and topic as a URL in the form:

172

mqtt(s):~/~/[username[:password]@]host[:port]/topic

173

-l : read messages from stdin, sending a separate message for each line.

174

-m : message payload to send.

175

-M : the maximum inflight messages for QoS 1/2..

176

-n : send a null (zero length) message.

177

-p : network port to connect to. Defaults to 1883 for plain MQTT and 8883 for MQTT over TLS.

178

-P : provide a password

179

-q : quality of service level to use for all messages. Defaults to 0.

180

-r : message should be retained.

181

-s : read message from stdin, sending the entire input as a message.

182

-t : mqtt topic to publish to.

183

-u : provide a username

184

-V : specify the version of the MQTT protocol to use when connecting.

185

Can be mqttv5, mqttv311 or mqttv31. Defaults to mqttv311.

186

~-~-help : display this message.

187

~-~-repeat : if publish mode is -f, -m, or -s, then repeat the publish N times.

188

~-~-repeat-delay : if using ~-~-repeat, wait time seconds between publishes. Defaults to 0.

189

~-~-quiet : (% style="color:red" %)don't print error messages.

190

~-~-will-payload : payload for the client Will, which is sent by the broker in case of

191

unexpected disconnection. If not given and will-topic is set, a zero

192

length message will be sent.

193

~-~-will-qos : QoS level for the client Will.

194

~-~-will-retain : if given, make the client Will retained.

195

~-~-will-topic : the topic on which to publish the client Will.

196

~-~-cafile : path to a file containing trusted CA certificates to enable encrypted

197

communication.

198

~-~-capath : path to a directory containing trusted CA certificates to enable encrypted

199

communication.

200

~-~-cert : client certificate for authentication, if required by server.

201

~-~-key : client private key for authentication, if required by server.

202

~-~-keyform : keyfile type, can be either "pem" or "engine".

203

~-~-ciphers : openssl compatible list of TLS ciphers to support.

204

~-~-tls-version : TLS protocol version, can be one of tlsv1.3 tlsv1.2 or tlsv1.1.

205

Defaults to tlsv1.2 if available.

206

~-~-insecure : do not check that the server certificate hostname matches the remote

207

hostname. Using this option means that you cannot be sure that the

208

remote host is the server you wish to connect to and so is insecure.

209

Do not use this option in a production environment.

210

~-~-tls-engine : If set, enables the use of a TLS engine device.

211

~-~-tls-engine-kpass-sha1 : SHA1 of the key password to be used with the selected SSL engine.

212

~-~-psk : pre-shared-key in hexadecimal (no leading 0x) to enable TLS-PSK mode.

213

~-~-psk-identity : client identity string for TLS-PSK mode.

214

~-~-proxy : SOCKS5 proxy URL of the form:

215

socks5h:~/~/[username[:password]@]hostname[:port]

216

Only "none" and "username" authentication is supported.

217

\\See https:~/~/mosquitto.org/ for more information.

)))

(((

and modify the /usr/bin/mqtt_process.sh script, the location to change the command option is below lines:

)))

(% class="box" %)

(((

//# Call MQTT Publish command

228

\\# 1. Case with User, Password and Client ID present (e.g. Azure)

229

if [ ! -z "$pass" ] && [ ! -z "$user" ] && [ ! -z "$clientID" ]; then

230

case="1"

231

mosquitto_pub $D -h $server -p $port -q $pub_qos -i $clientID -t $pub_topic -u $user -P "$pass" $C $cafile $PUB_FLAG "$mqtt_data"

232

\\# 2. Case with Certificate, Key and ClientID present (e.g. AWS)

233

elif [ ! -z "$certfile" ] && [ ! -z "$key" ] && [ ! -z "$clientID" ]; then

234

case="2"

235

mosquitto_pub $D -h $server -p $port -q $pub_qos -i $clientID -t $pub_topic ~-~-cert $cert ~-~-key $key $C $cafile $PUB_FLAG "$mqtt_data"

236

\\# 3. Case with no User, Certificate or ClientID present

237

elif [ -z "$user" ] && [ -z "$certfile" ] && [ -z "$clientID" ]; then

238

case="3"

239

mosquitto_pub $D -h $server -p $port -q $pub_qos -t $pub_topic $PUB_FLAG "$mqtt_data"

240

\\# 4. Case with no User, Certificate, but with ClientID present

241

elif [ -z "$user" ] && [ -z "$certfile" ] && [ ! -z "$clientID" ]; then

242

case="4"

243

mosquitto_pub $D -h $server -p $port -q $pub_qos -i $clientID -t $pub_topic $PUB_FLAG "$mqtt_data"

244

\\# 5. Case with User and ClientID present, but no Password and no Certificate present

245

elif [ -z "$pass" ] && [ -z "$certfile" ] && [ ! -z "$user" ] && [ ! -z "$clientID" ]; then

246

case="5"

247

mosquitto_pub $D -h $server -p $port -q $pub_qos -i $clientID -t $pub_topic -u $user $PUB_FLAG "$mqtt_data"

248

\\# 6. Case with User and Password present, but no ClientID and no Certificate present

249

elif [ ! -z "$user" ] && [ ! -z "$pass" ] && [ -z "$clientID" ] && [ -z "$certfile" ]; then

250

case="6"

251

mosquitto_pub $D -h $server -p $port -q $pub_qos -t $pub_topic -u $user -P "$pass" $PUB_FLAG "$mqtt_data"

252

\\# 0. Else - invalid parameters, just log

253

else

254

case="(% style="color:red" %)Invalid parameters" (%%)

255

logger (% style="color:red" %)"[IoT.MQTT]:Invalid Parameters - mosquitto_pub not called."(%%)

fi//

)))

= 4. Example to communicate to a simple MQTT server =

== 4.1 Overview ==

This section is an example to show how to set up LG01-N to communicate with a MQTT server. The MQTT server is a simple utility set up in a local PC. Note: User can set up same server via [[this instruction>>url:http://www.steves-internet-guide.com/install-mosquitto-broker/]].

265

266

267

== 4.2 Simulate via MQTT.fx utility ==

268

269

The [[MQTT.fx>>url:http://mqttfx.jensd.de/index.php/download]] is a MQTT client tool. We can use this to simulate a MQTT connection to our MQTT broker first to make sure the MQTT broker works. This will also help us understand how it works.

270

271

(((

272

In this test, the MQTT broker and MQTT.fx are installed in the same PC, so the MQTT server address in MQTT.fx should be localhost. Below shows how to connect to the server.

273

)))

274

275

[[image:image-20220527134929-6.png]]

276

277

Connect to MQTT Broker

278

279

280

After connected, use publish to public some thing to MQTT server. This to simulate upsteam

281

282

[[image:image-20220527135037-7.png]]

283

284

Upstream: Publish message to MQTT Broker

285

286

287

To simulate a downstream, use MQTT.fx to subscribe a topic, and publish something to this topic. as Below:

288

289

[[image:image-20220527135215-8.png]]

290

291

Downstream: Subscribe a topic to get downstream

292

293

294

== 4.3 Simulate via Dragino Command Line ==

295

296

For first try of MQTT connection, simulate via command line is recommend, there are many servers / connection type for MQTT. They are using different connection parameters. Simulating the connection via command line will help us rapidly connect to server and debug.

297

298

In the Dragino Gateway, we use [[mosquitto client>>url:https://mosquitto.org/]] for MQTT connection.

299

300

301

(% style="color:#4f81bd" %)**For Upstream**

302

303

command is [[mosquitto_pub>>url:https://mosquitto.org/man/mosquitto_pub-1.html]]

304

305

Example: mosquitto_pub -h 192.168.199.148 -p 1883 -t /channel/6543 -m temp=36

306

307

Note: 192.168.199.148 is MQTT broker address, the gateway and the MQTT broker PC are in the same network.

308

309

[[image:image-20220527135310-9.png]]

mosquitto_pub

(% style="color:#4f81bd" %)**For Downstream**

315

316

Use [[mosquitto_sub>>url:https://mosquitto.org/man/mosquitto_sub-1.html]] to subscribe the change on the topic.

317

318

[[image:image-20220527135440-10.png]]

mosquitto_sub

== 4.4 Configure Dragino UI for MQTT connection ==

324

325

This chapter are step by step to show to configure the Dragino Menu for MQTT auto connection.

326

327

(((

328

Go to (% style="color:#4f81bd" %)**Dragino Menu ~-~-> MQTT Client**

329

)))

330

331

[[image:https://wiki.dragino.com/images/thumb/b/bc/MQTT_Command_8.png/600px-MQTT_Command_8.png||height="249" width="600"]]

332

333

go to mqtt configure menu

334

335

(((

336

Select (% style="color:#4f81bd" %)**Forward to MQTT server**. (% style="color:red" %)**Notice**(%%): This option is removed from the latest firmware, in the latest firmware, if user submit "SAVE & APPLY" in MQTT page, the gateway will use MQTT service.

337

)))

338

339

[[image:image-20220527135742-11.png]]

forward to MQTT

=== 4.4.1 Configure the MQTT Client for Upstream ===

345

346

Below screenshot is same as the publish command:

(% class="box" %)

(((

mosquitto_pub -h 192.168.199.148 -p 1883 -i dragino-1b7060 -t CLIENTID/CHANNEL/data -m DATA

351

~/~/where the CLIENTID, CHANNEL & DATA are macro. represent for

352

~/~/CLIENTID: dragino-1b7060

353

~/~/CHANNEL: Remote ID in Channel settings; here is 78901 or 567456

354

~/~/DATA: The data stores in /var/iot/channels/

355

)))

356

357

[[image:image-20220527135828-12.png]]

358

359

MQTT Publish configure

360

361

362

[[image:image-20220527135929-13.png]]

363

364

MQTT Channel settings

(((

For example, if we put a data(temp=46) on the file /var/iot/channels/4567, because 4567 match the remote channel 78901. the gateway will run this command:

)))

(% class="box" %)

(((

mosquitto_pub -h 192.168.199.148 -p 1883 -i dragino-1b7060 -t dragino-1b7060/78901/data -m temp=46

)))

(((

to MQTT broker.

)))

(((

Below is a simulation to put this data to active the MQTT publish.

383

)))

384

385

[[image:image-20220527140023-14.png]]

MQTT Publish

=== 4.4.2 Configure the MQTT Client for Downstream ===

391

392

Below screen shot equal to this subscribe command:

(% class="box" %)

(((

mosquitto_sub -h 192.168.199.148 -p 1883 -i dragino-1b7060 -t command.

397

)))

398

399

[[image:image-20220527140100-15.png]]

MQTT Subscribe

(((

When MQTT broker receive a update on this topic, the gateway will get the update and use LoRa radio to broadcast this message. The LoRa parameters used for update is:

406

)))

407

408

[[image:image-20220527140148-16.png]]

409

410

LoRa Broadcast parameters.

411

412

413

And below is the subscribe simulation:

414

415

[[image:image-20220527140238-17.png]]

416

417

downstream simulation

418

419

420

== 4.5 Add LoRa support to communicate with remote sensor ==

421

422

In above section, we have configured the UI to support MQTT upstream and downstream. We can simulate via Linux command. In this section, we will guide how to communicate with remote LoRa End Node for upstream and downstream.

423

424

425

=== 4.5.1 Use LoRa Raw protocol for communication ~-~- For LG01/LG02 ===

426

427

(((

428

We can use [[LoRa Shield>>url:http://www.dragino.com/products/lora/item/102-lora-shield.html]] to send LoRa Raw data to Gateway and receive data from gateway.

)))

(((

The example Sketch for LoRa Shield +Arduino is here: [[LoRa_Shield_Sketch_For_MQTT>>url:http://www.dragino.com/downloads/index.php?dir=LoraShield/]]

)))

(((

And this link is the required library: [[arduino-LoRa-master>>url:http://www.dragino.com/downloads/index.php?dir=LoraShield/]]. Unzip this library and put in Arduino library location.

)))

(((

What does the Arduino Sketch do? The Arduino Sketch will:

441

)))

442

443

* Upstream: Keep sending a LoRa Message every minutes with this payload : <4567>tem=xx&hum=yy (Where xx and yy are temperature and humidity value generated randomly).

444

* Downstream: Listening broadcast message from gateway, and print it in console.

445

* The LoRa parameter settings in Arduino should match the LoRa settings in gateway, as below:

446

447

[[image:image-20220527140354-18.png]]

448

449

LoRa Parameter should match

450

451

452

Below is the test result after the Arduino Sketch is running.

453

454

[[image:image-20220527140459-19.png]]

Upstream Data Flow

[[image:image-20220527140542-20.png]]

Downstream Data Flow

=== 4.5.2 Use LoRaWAN Protocol for communication ~-~- For LG308/LPS8/DLOS8 ===

465

466

(((

467

Since firmware (% style="color:#4f81bd" %)**LG02_LG08~-~-build-v5.3.1585192026-20200326-1109,**(%%) Dragino LoRaWAN gateways support the communication to LoRaWAN (% style="color:#4f81bd" %)**ABP end node**(%%) locally without the need of LoRaWAN server. This feature allow us to integrate MQTT in the gateway to support LoRaWAN to MQTT forwarding or visa verse.

)))

(((

When use test this feature, please use the version higher then : (% style="color:#4f81bd" %)**LG02_LG08~-~-build-v5.4.1593400722-20200629-1120**(%%), in this version, the upload format is changed and readable, which is easier for integration.

472

(% style="color:#4f81bd" %)**Video Instruction**(%%): (% style="color:blue" %)[[https:~~/~~/youtu.be/qJTY441-t90>>url:https://youtu.be/qJTY441-t90]]

)))

(((

(% style="color:#4f81bd" %)**Step 1**(%%): Refer [[Communicate with ABP End Node>>doc:Main.Communicate with ABP End Node without LoRaWAN Network Server --- LG308.WebHome]] to know how to set up LG308 to work with LoRaWAN End node.

)))

(((

(% style="color:#4f81bd" %)**Step 2**(%%): Make sure your Radio settings match the End Node settings.

481

)))

482

483

[[image:image-20220527141235-21.png]]

484

485

Use Same Frequency Band as End Node

(((

(((

(% style="color:#4f81bd" %)**Step 3**(%%): Set up publish format and MQTT channel. The LG308 will store the Data from End node in (% style="color:#4f81bd" %)**HEX format**(%%) in the file. And we need to config the format to (% style="color:#4f81bd" %)**META**

)))

)))

[[image:image-20220527141450-22.png]]

495

496

Publish mush use META as data format for LG308

497

498

499

(% style="color:#4f81bd" %)**Step 4**(%%): Map the Device Address to Remote ID in MQTT server.

500

501

[[image:image-20220527141727-23.png]]

502

503

Map Dev Addr to remote ID

504

505

506

(% style="color:#4f81bd" %)**Step 5: Upstream**(%%): Save the change, we can see the log info via "sytem log", End Node and MQTT Server

507

508

[[image:image-20220527141843-24.png]]

Upstream Process

[[image:image-20220527141933-25.png]]

Choose ASCII Format

[[image:image-20220527142028-26.png]]

LHT65 Decoder

(% style="color:#4f81bd" %)**Step 6: Set up subscribe**(%%)**:** Subscribe a topci for downstream.

524

525

[[image:image-20220527142115-27.png]]

Subscribe to a topic

(% style="color:#4f81bd" %)**Step 7: Downstream**(%%): Save the change, we can see the log info via "sytem log", End Node and MQTT Server.

531

532

[[image:image-20220527142239-28.png]]

Downstream Flow

(% style="color:red" %)**Notice: The text use for Downstream must meet the requirement from [[LG308 Downstream Payload>>http://wiki1.dragino.com/xwiki/bin/view/Main/Communicate%20with%20ABP%20End%20Node%20without%20LoRaWAN%20Network%20Server%20---%20LG308/#H2.2Downstream]]**

537

538

539

= 5. Example For Different MQTT Servers =

540

541

(% style="width:731.222px" %)

542

|=(% style="width: 270px;" %)(((

543

[[image:image-20220527142706-30.png]]

544

545

== [[ThingSpeak Server>>url:http://www.thingspeak.com/]] ==

546

547

548

[[Examples>>url:https://wiki.dragino.com/index.php/ThingSpeak_Examples]]

549

)))|=(% style="width: 242px;" %)(((

550

[[image:image-20220527142733-31.png]]

551

552

553

== [[乐联网平台>>url:https://www.lewei50.com/]] ==

554

555

556

[[lewei Example>>url:https://wiki.dragino.com/index.php/Lewei_Example_mqtt]]

557

)))|=(% style="width: 214px;" %)(((

558

[[image:image-20220527142820-32.png]]

559

560

== [[AWS-IOT>>url:https://aws.amazon.com/cn/iot-platform/how-it-works/]] ==

561

562

563

[[AWS Examples>>url:https://wiki.dragino.com/index.php/MQTT_Forward_to_Amazon_AWS-IOT]]

)))

= 6. How to Debug =

User can login the gateway's console and run (% style="color:#4f81bd" %)**logread -f**(%%). It will shows the output when there is packet arrive.User can see if it is correct.

569

570

571

= 7. How to ask for Support =

572

573

If a user still not have trouble making it works. please send a mail to support@dragino.com with the below info:

574

575

* Detail of your LoRaWAN end node.

576

* Gateway Model and firmware version