Published Data

Publish Data Published data is the CloudPlugs IoT Data Lifecycle Management application. It provides a view into the timeseries database of all the data published by Prototypes, Production Things and Controllers without having to build queries. It also allows to:

  1. Manage all the device data and the pub-sub channels.
  2. Publish new data with retention policies.
  3. Delete data and channels.
  4. Export data in CSV format.
  5. Visualize the data in various forms.

The Published data management console has two tabs:

  1. Data to manage the published data lifecycle, and
  2. Data Views to visualize the data published by Prototypes, Production Things and Controllers in various formats.
Data Data View

Data

The Data tab is used to manage the lifecycle of published data. It has two subpanels:

  1. A Channels Management Panel to display and remove data channels.
  2. A Data Management Panel to create, edit, delete, export and to perform bulk actions on data.

Channels Management Panel

The Channels panel allows you to visualize and navigate your channels in a tree structure. You can perform three functions:

  1. Select a channel or subchannel by navigating through the channels tree to display and manage its data.
  2. Reload the channel list clicking the Reload ( ) button.
  3. Delete a channel, a subchannel or a channel with all its subchannels clicking the Delete ( Delete ) button.

If data is published in a new channel and it is not displayed in the Published Data Console, reload the channels list and select the new channel to see its published data.

WARNING!

When a channel is deleted all its sub-channels and data are also deleted.

Back to Top

Data Management Panel

The Data Management Panel allows the lifecycle management of the published data. The panel enables to:

  1. Filter data published based on multiple criteria available through the column title dropdown menus.
  2. Publish new data (New data).
  3. Open and edit (Open) the published data.
  4. Delete (Delete) data.
  5. Export (Export) data.
  6. Perform bulk actions (Bulk Actions) on the data published by all the devices in your account.

The Data Management Panel columns display:

  • Data ID - the unique identifier for every data point published in the platform.
  • Publisher - entity which published the data. It may be a device (Prototype, Thing or Controller), or the Company if you are publishing as the Company in your CloudPlugs account.
  • Channel is the channel where the data was published.
  • Data is the JSON that was published.
  • Published is the time stamp of when the data was published.
  • Expires indicates the data retention period set for the published data using the ttl (time to live) function.
Data Panel

Back to Top

Filter Published Data

It is practical and many times necessary to view only certain data. The platform provides several ways to filter data.

The data published can be filtered by channel or sub-channel by selecting the source in the Channels Management Panel.

In addition, published data can be filtered using the data management panel’s column filters as follows:

  • Publisher - displays only the data published by the selected Publisher.
  • Channel - displays only the data published in this channel.
  • Published - displays data published within the time period specified.
  • Expires - displays data with expiration dates within the time period specified.

The example below shows a filter to display data that expires on March 25, 2018 between 15:30 and 17:45 and the result.

We first set the From date and time:

Data Filter 1

We now set the To date and time:

Data Filter 1

The result when the filters are enabled by clicking on the selection boxes of the filters is shown below.

Data Filter 1

Back to Top

Publishing Data

This functionality allows to publish data impersonating a device (Prototype, Thing, Controller). It is used to test specific device or application behavior when certain data is received by the platform.

New data

Data may be:

  1. Numeric positive or negative.
  2. Alphanumeric.
  3. Structured JSON.
  4. Location coordinates in a JSON structure. This allows to track mobile devices. See an example below.

There are two options to publish data:

  1. Using the Published Data application on the CloudPlugs IoT web desktop.
  2. Using the Publish Data panel on a specific Prototype, Thing or Controller device.. The panel is opened through the Publish Data tab ( New )

In both cases, to publish data:

  1. Open the Publish Data window with the New Data ( New ) button in the Data Panel.

  2. Select the Publisher which can be one of:

    • Company - a universal publisher that is not associated with any device and whose data published is visible to all devices.
    • Prototype - a Prototype device.
    • Thing - a Production Thing device.
    • Controller - a Controller device.
    • Plug-ID - the Plug-ID of the device that will publish data.

    An additional selection dropdown menu allows to choose an existing Prototype, Thing or Controller. If the Publisher is Company or Plug-ID, the Plug-ID of the device publishing data is automatically filled in.

  3. Enter the target channel name. If an existing channel name matches the channel as it is entered, it will be suggested to you.

  4. Enter the data which can be:

    • Positive or negative number or numbers. If more than one number is entered then:
      1. Only the last number is accepted if they are separated by commas (e.g. enter 1,2 and only 2 is accepted). Same happens if you enter 1, in one line and 2 in the line below.
      2. Numbers entered in separate lines without a comma separator are accepted (e.g. entering 1 in one line and 2 in another results in data “1 2”.
      3. Decimal numbers are accepted.  For example, 1.2 is accepted as 1.2.  1.2, 1.3 results in 1.3 being published.
    • Alphanumeric data. You can publish data such as “abc”, “1 23 ab c”, “1;3;x;y;z” etc.
    • Structured JSON objects. JSON objects have the syntax:
{
	"internal": 70,
	"external": 90
	}
  • Location coordinates, which have the syntax:
    {
    	"x": 70.1234578676,
    	"y": -65.098080890
    	}

    where x is the longitude and y is the latitude relative to Greenwich.

  1. Select the date and time of publishing. The default is now. Back dates are allowed.

  2. Select the expiration date and time of the data. Only future times are allowed. Data can live as little as one (1) minute and it can live up to 6 months. The default time-to-live ( ttl ) for data is 6 months.

  3. Save and Close (Save and Close) to see the data displayed on the Data panel and on the chart.

The following images display the publish action of Prototype “Test1” of temperature value 70 on channel test1/temp, and the result of multiple publish actions on the same channel. To display the chart, we simply selected unsaved under Views/Charts.

If some element of the data like the expiration information is not showing, simply refresh the panel with the refresh ( ) icon on the bottom bar.,

Generate Test Data

Back to Top

Edit Published Data

Data that has been published can be edited as follows:

  1. Select and Open the published data of interest by double clicking on it or using the Open (Open) button.
  2. Modify the expiration date to a past or future time period.
  3. Modify the data itself with any valid JSON.
  4. Save (Save) or Save and Close (Save & Close).

The images below show a published data before and after its expiration data and value have been modfied.

Data 1 Data 1

Back to Top

Export Published Data

The data published in a channel can be exported into .CSV files. JSON structures will be exported as such.

IMPORTANT!

The export action will export all the data existing in the selected channel. A single export action is limited to 500,000 data records and 32MB in a single request.

To export the data of a particular channel:

  1. Select the channel or sub-channel from the Channels Panel.
  2. Click on the Export (Export) button.
  3. Open the .CSV file downloaded to confirm that the data has been properly received.

Back to Top

Bulk Actions

The Bulk Actions button allows to perform actions across multiple Prototype and Production Thing devices simultaneously. Production Things subject to the action are selected through their Production Template and serial numbers.

There are two types of actions possible:

  1. Delete data.
  2. Replace Data values with new data.

Here are examples of replace and delete values bulk actions on channels enocean/4/status and test1 on data from a specific date range.

The replace action is performed on a Production Thing belonging to Template CP-Demo. The device’s serial number is contained in the 20 serial no.csv file. Serial numbers must be uploaded using a valid CSV file or the operation will fail.

The delete action is performed on data in channel test1 from Prototype Test1.

Please note that these actions are not executed in real time and are completed by CloudPlugs IoT as a background task.

Bulk Replace Values Bulk Delete

Back to Top

Data Views

The Data Views tab is used to visualize published data through programmable charts with multiple layout options. The charts are updated in real time as data is published in the platform. By default, the platform creates and updates line charts of numeric data is published.

With Data Views you can:

  1. Create data views
  2. Open and edit data views
  3. Delete data views.

There are two ways to create Data Views:

  1. Using a device’s Publish Data panel ( Pub data ).
  2. Using the Data Views panel of the Published Data application of the CloudPlugs IoT web desktop.

Creating a Data View from a Device’s Data Panel

If the device has existing data on a channel, an unsaved data view is automatically created. If the data is numeric, selecting the view will automatically display a chart on the chart panel of Data Views. See example of the data of a Prototype below.

Published data

The data view can then be opened, edited and saved if desired.

To open and edit a data view:

  1. Open the unnamed view by double clicking on it or by clicking on Open ( Open ).
Data View 1
  1. Enter a Name for the Data View.
  2. Select a layout. Multipanel layouts allow you to create multiple views of the data from one or multiple channels. Notice that by default, the chart is showing the two fields (temp and humidity) in the channel. In this example, we will create individual charts for each one on 2 separate panels.
Data View 2
  1. Click on the panel you want to configure, or on the gear button that shows when you hover over the panel and enter the required data:
    • Publisher - a Prototype, Thing, Controller, a device identified by a Plug-ID, any device or Company. The data published by Company is visible to all devices.
    • Channel - from which the data will be read.
    • Data Field - if the data is a JSON with data fields
    • Title - gives a title to your chart
    • Select the type of chart which can be a Line, Bar, Gauge or Map chart.
      • Line charts allow color and fill selection.
      • Bar charts allow color selection and Map charts only start date and period. Gauges have their own set up.
    • Enter the starting date for the chart and the period that you want to display. These parameters do not apply to Gauges.
Data View 3
  1. Save and Close ( Save and Close ).
Data View 4
  1. Repeat for the other value(s) to be charted.
Data View 5
  1. The finished view looks like:
Data View 6
  1. Save and Close ( Save and Close ).

The result in the Thing’s data panel is shown below. Notice that due to space limitations, only the temp data is displayed.

Data View 7

The full Data View can be found opening the Published Data ( Published Data ) application and selecting Data Views as shown below.

data view 8 data view 9

In addition, the following actions can be made on Data Views:

  1. The Data View panels can be edited by clicking on the panel’s gear icon.
  2. Time based charts allow:
    • Scrolling over time periods with the forward (>) and backwards (<) arrows.
    • Zoom into a period. Simply position your pointer on the desired date and drag a rectangle over the period you want to zoom into. To unzoom click on the (Unzoom) button.

Here are two examples for creating numeric and location based data views.

Back to Top

Creating and Editing a Data View from Published Data

The process for creating data views using Published Data -> Data Views is similar to the process described above.

  1. Open the Published Data Application ( Published Data ) from the web desktop.
  2. Select the Data Views tab.
  3. Click on New ( New ).
  4. Follow the process described above.

Let’s now edit the view we created above, change the layout to 3 panels and add a gauge.

  1. Open the view from Published Data->Data Views.
data view 8 data view 9
  1. Change the layout. We will use layout 3C.
Data View 10
  1. Add a gauge. We configure it as follows and save the configuration.
Data View 11
  1. Here is the resulting data view and we save it.
Data View 12

Location Data Views Example

This example shows how to create a data view based on location data of a mobile device.

  1. Publish location data in a channel that we call location as shown below. As the data gets published the trace of movements is shown on the Views / Charts panel.
Location 2
  1. Double click on the unsaved Data View to open it and set its properties.

  2. Enter a Name and select a Layout. For this example, the Layout will be 1.

  3. Click on the configuration gear, give the map a Title and set up the time period that will be charted.

  4. Save (Save) or Save and Close (Save & Close) the new Data View. The result is shown below.

Location 2

Back to Top

Delete a Data View

To delete a Data View:

  1. Open the Published Data Application ( Published Data ) from the web desktop.
  2. Select the Data Views tab.
  3. Select the Data View
  4. Click on Delete ( delete ) from the menu ribbon.
Still need help? Get in touch!
Last updated on 26th Mar 2019