Modbus TCP network problem

jconnally · Post by **jconnally** » 11 Jul 2019, 16:21

Hello,

Have any TCP/IP networking problems been reported for the RevPiConnect? Specifically, has anyone reported the Pi acknowledging a query for data but never actually responding with the data without the requesting host initiating a TCP Retransmission? I am running a RevPiConnect as a Modbus gateway device and am seeing this issue. A host PC running process control software is on the TCP/IP side of the gateway and a number of RS485 slave devices are on the other side. All slave devices have been independently verified to consistently respond to read/write requests within 20 msec or less without failure.

Attached is a snapshot of data from WireShark running on the PC host. The eth1 interface of the RevPiConnect is configured to have a static IP address of 192.168.0.101. The host PC has static address of 192.168.0.1. This network is local to the PC and RevPi. No other devices exist on the network other than network switches. Beginning at line item 4, the PC host issues a query to the Pi which responds with an acknowledgement [ACK]; however, more than 4 seconds elapse (4.364) without a response which exceeds a timeout in the host PC software. The PC then issues a second query which initiates a TCP Retransmission. After that, the RevPi responds with an acknowledgement [ACK] followed by a duplicate acknowledgement [TCP Dup ACK 8#1], then the response data. This behavior is exhibited somewhat randomly once every 24 to 36 hours and is causing a problem for our host PC control software. Normally, the initial query from the host is immediately (within 50 msec or less) followed by a response from the Pi with valid data. The packet data of the initial query triggering the retransmission has been verified to be a valid Modbus request. Also, the Modbus protocol we are running logs all protocol failures including slave device timeouts to a log file. The log file is clean and shows no errors.

Can anyone provide any explanation for this behavior?

Regards,
JConnally

Post by **dirk** » 17 Jul 2019, 11:33

Dear JConnally, thank you for the detailed description of the error. This helps a lot. I can see some suspicious unit-ids in the Wireshark trace. Which hardware setup do you have and which image do you use?
Please post the /etc/revpi/_config.rsc file and the output of

Code: Select all

cat /etc/revpi/os-release

. Can you describe your hardware setup a bit more in detail? I have understood that you use the RevPi Modbus Slave on 192.168.0.101 and a Modbus Master (which?) on 192.168.0.1, richt? Why are there sometimes different unit-ids, i.e. 64 and 1?

jconnally · Post by **jconnally** » 18 Jul 2019, 23:13

Hi Dirk,

There is no os-release, just image-release. cat /etc/revpi/image-release gives 2018-07-17-revpi-stretch. The config.rsc is attached. The RevPi is used as a TCP to RS485 gateway. The RS485 side has two slave devices with Modbus ID's of 1 and 64 respectively. The value 1 is an RS485 Wago and 64 is a custom realtime board called the "comm board". It's job is to communicate via another RS422 network with a set of brushed motors. The RS485 baudrate is 38400 bit/sec, 8 bits, 1 stop bit, no parity. Overall, we are not having difficulty with the RS485. On the TCP/IP side, Port A (interface eth0) of the RevPi connects to the corporate lan and is configured for dhcp. Port B (interface eth1) is statically configured to IP address 192.168.0.101 and connects to a switch on a local device network. The static eth1 address is set in /etc/dhcpcd.conf. The PC has a dedicated network card with IP address 192.168.0.1 and connects to the device network switch. The PC runs DeltaV software which provides supervisory control of the RS485 devices; the gateway simply passes on the read/write commands received over Modbus TCP to the RS485 side. Every so often, DeltaV reports an Maintenance Ethernet IO alarm. The alarm corresponds to what we see in WireShark. A consistent pattern emerges each time: the device is queried, the query is acknowledged, but no data is sent. A timeout in DeltaV expires which triggers the alarm. DeltaV then sends a second query which is acknowledged by the RevPi followed by duplicate acks and the data. At this point, it looks like a problem in the RevPi network stack or possibly in the driver layer. Any help in how to trouble shoot this problem would be appreciated.

Regards,
John C.

zhan · Post by **zhan** » 22 Jul 2019, 14:19

Hi jconnally,

I am on this issue now.
Firstly I am analyzing the code of transceiving. Could you provide the kernel log when the error happened?
fyi. command dmesg to get the kernel log from booting, which is archived in /var/log/kern.log.*.xz

And is the software you run in PC is available to me, if so, could you please tell me the name, I download it from the internet and have a test of this.
Or could you provide me the pcap file ? by email or ticket system. you can erase some information in data field with packet editor tools as you need.

Further, I would like to ask could you run a debug purpose version from me?
or I provide you the debugging code patch, and you get the source code from github and apply the patch, the compile, install and run it ?

Best regards,
Simon

jconnally · Post by **jconnally** » 22 Jul 2019, 18:07

Hello Simon,

Thank you for your assistance. Attached is the WireShark log file. The Software we run on the PC is DeltaV overlaid with Thermo Fisher's TruBio Sofware. DeltaV and the PC are Emerson Corporation's products and TruBio is a Thermo Fisher product so I don't think this will be a practical for you to run this unless we send you a complete system with licensing dongles. We'll retrieve the kernel logs and get those to you. In the meantime, please send on the patch and detailed instructions on how to apply it, the github URL, plus download and install instructions for the build and debug tools.

Regards,
John C

jconnally · Post by **jconnally** » 23 Jul 2019, 02:08

Simon,

Attached is a zip file of the kernel logs of one of the Kunbus Rasp. Pi's that had the TCP/IP communication problem. I expect more to follow soon.

Regards,
John C.

zhan · Post by **zhan** » 23 Jul 2019, 11:11

Hi John,

in the config.rsc, there is no Modbus device configured.
Have you used other Modbus tools in revpi connect device, maybe ?
If you are sure about that, could you send me the topology and all relavent configurations on revpi connector?

Regards,
Simon

jconnally · Post by **jconnally** » 23 Jul 2019, 19:34

Hi Zhan,

We are running our own Modbus protocol purchased from Proconx Corporation. FieldTalk Modbus master and Slave C++ available at https://www.modbusdriver.com/shop/index.php?cPath=48.

The reason we use this software is to handle a known issue with one of our RS485 devices; essentially, it involves handling a value exception error in response to a valid read registers request. Off the shelve commercial modbus software would generate an error which we don't want. Beyond that we are running the RevPi "out of the box" with Debian stretch.

Regards,
John C.

zhan · Post by **zhan** » 24 Jul 2019, 10:20

Hi John,

what you mean issue of RS485 ? And what is the error generated you don't want ?
As many components(FieldTalk, Deltav, ...) are little by little be aware by us, we are now confused some how,
so could you make a graph(hardware connections, protocols be used, etc) to show your scenario to us.
Otherwise it hard for us to help you.

Regards,
Simon

jconnally · Post by **jconnally** » 24 Jul 2019, 18:08

The issue with the RS485 device is that it responds to a valid Modbus read register request with a value exception response. We know is this an error and handle it in the code that services the read request on the RS485 side. The problem we are trying to solve is unrelated to the RS485 devices. The TCP/IP side of the Kunbus does not respond with data when requested. It looks like a problem in the TCP/IP stack or more likely at the hardware level. The kernel logs provided show an issue with the smsc95xx at times. At this point, we would like to proceed with building a debug kernel and get more insight as to what is happening within the TCP/IP stack of the RevPi. In the meantime, I will provide a diagram of the system.