Testing MQTT with Telemetry with arbitrary function

Do you need to test your IoT platforms against arbitrary telemetry data from
a variety of sensors?

You can easily to generate ANY data you want from ANY sensor you want
using MIMIC IoT Simulator.

The below screenshot shows a NODE-RED subscriber client graphing a
simulated sensor publishing data of a Sine function with configurable
amplitude. In this case we just kept expanding the amplitude every minute.

The code to achieve that is shown and consist of a total of 10 lines including
comments and the effort in minutes.

With the same code, you could have multiple sensors sending the same
function, but different amplitudes at any point in time. Any other
parameters can be made unique for each simulated sensor.

The next screenshot shows 4 such sensors with different amplitudes
simultaneously. Any more would clutter the single graph too much.

Video: Monitor end-to-end response time of your IoT Application with 10k Sensors

This 5-minute Youtube video shows how to monitor response time to a
MQTT broker in an IoT Application with 10,000 active publishers.


This is not only important in the selection process, but also for ongoing
monitoring / troubleshooting, as outlined in our previous blog post
"IoT Sensors Need to be Managed", and this Gartner report.

We are following the testing methodology outlined in our previous post
"MQTT performance methodology using MIMIC MQTT Simulator" to
minimize the interference between the test equipment and the system
under test.

We are using the open-source Node-RED flows published in our Github
repository to measure and graph end-to-end latency between a publisher
and subscriber once a second for several brokers.

This simulates the latency between your sensors that are publishing
telemetry and your application that is consuming the telemetry.

We are using MIMIC MQTT Simulator to deploy 10,000 publisher clients
to the MessageSight broker on our intranet.

Online MIMIC MQTT Lab for Samsung Artik

We invite you to check out this free, online MIMIC MQTT Lab for the
Samsung Artik IoT platform, available on the Artik marketplace.

In 3 minutes you get a virtual lab with a simulation of a large-scale IoT
control system based on the MQTT standard.

In this lab you can explore:

• authentication and access of MQTT-based sensors to Artik in your own account
• telemetry flowing from sensors to defined device types in Artik
• charting of that telemetry based on device manifest
• control of actuators from defined rules in Artik

Check out this Youtube video for the entire process from an empty Artik
account.

MIMIC MQTT Simulator and Elk Stack

Thinking of feeding MQTT messages of your sensors into an
Elk Stack via a MQTT input plugin?

You could deploy hardware sensors or test clients for your testbed.
Alternatively, MIMIC MQTT Simulator provides a large virtual sensor
network of up to 100,000 sensors per host, so you can rapidly prototype
different types of sensor scenarios and scale up cost-effectively.

MQTT performance methodology using MIMIC MQTT Simulator

Performance of small-scale environments never predicts behavior of
large-scale deployments. But, it is too expensive to setup large numbers
of your MQTT sensors to test your IoT back-office platform, including
MQTT broker and client applications.

With MIMIC MQTT Simulator, it is simple to create large sensor simulations
to verify performance. The methodology is to use MIMIC to simulate a
large environment with synthetic background throughput, then verify
your performance requirements (eg. maximum round-trip delay) either
with a small number of your real-world sensor, or with another MIMIC
setup measuring end-to-end latency. That way you are sure the synthetic
load is not impacting your measurement setup.

In the screenshot above we are running 10 sensors with an end-to-end
measuring instrumentation, and the end-to-end delay is graphed in the
bottom graph. It shows minimum, average and maximum delay for messages
from those sensors to a subscriber running in the same MIMIC.

From another MIMIC instance, we keep adding a synthetic load onto the
MQTT broker under test, from 0 to 1000 in steps of 100. The upper graph
shows the size of the background load over the 15 minutes of the test.
Each background load sensor publishes at 1 message per second, so the
throughput is the same as the number of sensors. This is trivial to
change in MIMIC to conform to your real-world expectations.

As you can see, the delay is only slightly increasing over time, except
for 2 notable bumps at 600 and 1000 sensors. It is trivial to repeat the
scenario, and verify that indeed there is a reproduceable problem. You
would never know if you did not do the tests.

Test your management application with hundreds of Dell iDRACs

Testing your configuration management application or data center
orchestration software for large scalability requires you to setup a
large lab with many servers.

In the case of Dell servers, the Dell iDRAC is a card with out-of-band
control and monitoring of a Dell server. Rather than buying many servers
and hardware, you can simulate the iDRAC and server in MIMIC Web Simulator.

The screenshot below shows the Dell OpenManage Essentials application
managing 100 simulated Dell servers. The simulation allows complete
control to achieve whatever testing or demo scenario you want.

Once simulated in MIMIC, each of these devices are available on
the network as if they are real devices. Your management applications
can access them and have a complete control over their configurations.
You can easily start, stop them and change their parameters at
run time. You can customize the simulation at run-time to introduce
faults.

SNMP Simulators: build vs. buy

Faced with the prospect of simulating large network management
environments, the typical build vs. buy dilemma emerges: should you pay
for a commercial solution, or build one of your own?

There are several free open-source simulators available, promising to tilt
that equation in favor of "build": since a seemingly viable alternative is
already available, you'll just modify what's out there to suit your needs.

This will work out in only the fewest scenarios: the simplest small-scale
simulations.  For most other uses, there are many pitfalls:

1. Meeting the requirements: You have to evaluate the best open-source
for your purposes, and you risk making a sub-optimal choice. You might
pick one, only to find out it does not handle your changing requirements;

2. Developer training: You have to train your developers on the chosen
solution(s), not only from a end-user perspective, but down to the nitty
gritty design, source code and build procedures. You are essentially
investing in something other than your core competency;

3. Quality: Open-source might be high-quality for commodity software
(Linux, OpenOffice), but in highly-specialized areas it usually has few
developers/maintainers, goes out of date, and results in lower-quality
software.

4. Scalability: Designs that work for a small number of simulated entities
usually will not for large numbers due to worse-than-linear behavior of
algorithms. This will usually result in unacceptable performance.

5. Beyond SNMP: Network management applications these days use more
than SNMP to access the managed entities. This requires an integrated
solution that simulates not only the SNMP interface, but also command-line
interfaces (Telnet, SSH), flow-based protocols (NetFlow, sFlow), Web services
(XML, SOAP, REST, etc).

6. Budget for software maintenance: The initial development task is
usually a small part of the total effort. At least one developer has to
continuously maintain and upgrade the software, post messages on some
user groups for getting it built and deployed but also try to get timely
answers for resolving bugs.

 The end result is usually that you'll spend way more effort on the
"build" decision than the cost of the commercial solution would have
been to start with, impacting time-to-market, quality, support, etc.

Many large companies have already chosen to buy instead of spending
effort re-inventing the wheel. Their thinking is that it is better
to spend time and energy behind the companies' core product and services.
Do they know something you don't?

MIMIC SNMP Simulator has been on the market for 20 years. Simulators is all
that Gambit does and has hundreds of man years invested in. The end result
is a mature, high-quality, high-performance simulator that runs at most of
the large networking vendors world-wide. For example, reference customers
are vendors such as Lancope and Xirrus, and end-users such as Pepco and
SITA.

Combined with the other simulators in the MIMIC Simulator Suite you can
have a large realistic environment for development, testing, demo and
training of your network management application.

MIMIC runs on physical servers or the cloud and has been adapted to many
unforeseen uses. For example, Cisco runs MIMIC in combination with Prime
Infrastructure in their on-demand cloud platform dCloud.