Cisco Data in Motion Application Programming Interface Reference Guide

This document provides the information that is required to understand and use the Cisco Data in Motion (DMo) RESTful application programming interface (API).

This document is intended for developers who want to use the Cisco Data in Motion RESTful API to control various DMo features and capabilities and to access DMo resources and data. It assumes the developers have knowledge or experience with Cisco Data in Motion, a web-level programming language, the Getting Started Guide, and the following:

• Representational State Transfer (REST) Architecture
• JavaScript Object Notation (JSON)
• Web Services
• Hyper Text Transport Protocol (HTTP) and Secure HTTP (HTTPS)
• Web Distributed Authoring and Versioning (WebDAV)

Introduction

In the evolution of the Internet what is being termed as the Internet of Things (or IoT) refers to the unique identifiable physical objects and their virtual representations interconnected in a vast network environment. It is also referred to as Internet of Everything (IoE) within certain contexts with distinctions between the two concepts. Cisco defines IoE as the networked connection of people, process, data and things. In contrast, IoT only involves the networked connections of physical objects and data representations and does not include the people and process components. Hence, IoT is a subset of IoE, where IoE comprises of multiple dimensions of technology transitions, including IoT. Today, more than 99% of things in the physical world are still not connected to the Internet. As sensor devices and nodes are attached to the Internet, they will generate vast amount of data that will need to be processed. The amount of data generated will dwarf the already huge amount of Internet traffic generated today. From research predictions, more than 30 billion devices will be connected to the Internet by 2020.

To address these forthcoming IoT and IoE challenges, Data in Motion (DMo) is a Cisco software technology that provides data management and first-order analysis at the network edge. The primary objective of Data in Motion is to be built into product solutions by Cisco, Cisco customers and partners. DMo rethinks the way data is acquired and managed. It instructs edge devices, such as sensors and sensor gateways as to what data is of interest and what aspects of it to capture. From a user's perspectives, not all data is of interest and the ability to set rules and policies on the edge devices along with capabilities to search the data in real-time and trigger subsequent context-aware actions will inspire powerful technical advantages. The current underlining technical approach is to "store first, analyze later" where all the data is processed in the cloud and backend servers at a later time may not be feasible not only due to the large amounts of data; but also the need to take actions in real-time based on the streaming data (i.e. on data that is in motion and not static). Hence, the key benefits of Data in Motion would be dramatic bandwidth reduction as we reduce the amount of data being sent back by eliminating data that is not needed. Secondly, devices at the edge may have the ability to understand the data and the ability to query the devices for semantic information available in sensor data will be invaluable.

The technical approach towards evangelizing Data in Motion to the masses is to provide an Eclipse Foundation Application Programming Interface (API) for building it into an application ecosystem by Cisco partners and customers. DMo is configurable by these applications to analyze data as it enters and travels over the network. Examples of the analysis can include: (1) finding specific data of interest, (2) summarizing the data in a defined way, (3) processing the data to generate new result data and other application defined analysis. DMo is designed to be deployed and to operate in a distributed and heterogeneous manner in order to manage and control the flood of data from IoT and IoE sensors and data sources.

Another advantage of DMo is to offer different processing and analysis at each location within the network. DMo targets many different heterogeneous Cisco platforms, such as c819 ruggedized router, Connected Grid Router (CGR), Wireless Access Point, processing blade in a router and in a Virtual Machine (VM) in an Unified Computing System (UCS) and within a data center. The many instances of Data in Motion in the network will work together to deliver an integrated data analysis "fabric" for large-scale IoT and IoE.

Overview

A top-down system overview is to consider an architecture based on a publish/subscribe mechanism where rules and policies are registered on devices/gateways that have visibility into and communicate with sensors. The rules can be used to describe what data should be acquired. For example, we can acquire data based on network parameters, such as protocol, IP address or port number. The rule can also specify that content payload that matches certain criteria should be processed or extracted. For example, a rule can attempt to acquire data from sensors where the temperature is within a certain range.

The provided API interfaces with the user's programming environment. In essence, a user writes a software program that specifies what data s/he is interested in. The API helps the user translate rules in open standard JSON format encapsulated as a REST message that can be understood by the API.

A key part of the system overview is the format of the JSON messages used to express a policy. The API to the edge device of interest using a RESTful communication paradigm then sends this policy. This is the publish part of the architecture.

An edge device supporting the API will listen to the REST messages containing the JSON payloads that express the rules and will register them. A component of the system runs on the edge device and translates the rules from JSON format to the internal format of the device. The device will be responsible for translating the JSON messages into internal representations that it can understand since these representations are specific to each device. Traffic that flows through the device will be searched against the rules. The devices may have the ability to index and search the payload and content in the sensor data. It also can execute ad hoc queries on the payload. In this manner, the data at the edge of the network can be searched in real-time using the API to discover nuggets of information from the mountain of raw data. The rules may also specify what should be done with the matching traffic. Results of successful hits could be sent (again in a RESTful manner) to an endpoint, which will be listening for responses from the edge device. This is the subscribe part of the architecture.

The DMo architecture provides users the capability to transform the raw data generated by sensors into information. DMo provides RESTful APIs for users to communicate any edge supported edge devices allowing users to write their own applications to acquire data and events.

A typical user will be interested in transforming data from multiple edge devices generating different types of data into events of interest that can be analyzed by more powerful (usually domain specific) analytics engines in the cloud.

The rules can be used to describe what data should be acquired. Data acquisition can be based on network parameters such as protocol, IP address or port. It is also possible to specify that content payload that matches certain criteria should be processed. For example, a rule may be created to acquire data from sensors where the temperature is within a certain range.

Once the rules are pushed to the edge device using the APIs, any traffic that flows through the device will be searched against the rules. DMo allows the edge device to have the ability to index and search the payload and content in the sensor data and also to execute queries on the payload.

In this manner, the data at the edge of the network can be searched in real-time using DMo to discover information from raw data. Once the information is discovered users will typically send either the whole or part of this information to an endpoint to perform further operations.

Features of DMo

Management using RESTful API with webDAV extension

Dmo uses RESTful service API with webDAV extension for managing rule resources. Rules can be install using RESTful API with a JSON over HTTP data format.

Filter by network parameters

DMo is able to filter traffic based on packet information from the LINK, Network and TRANSPORT layers. This includes filtering based on Protocol: TCP/UDP/HTTP and filtering based on source/destination MAC address, IP address, port number. Multiple IP and mac addresses can be grouped together using subnet masks and multiple ports can be specified with a range.

Filter by HTTP header

If the traffic stream contains HTTP, the user is able to apply filters using HTTP headers such as "Content-Type", "Content-Length", etc.

Filter by payload content

Dmo is able to parse and filter based on the payload content. This is applicable to both raw tcp/udp payload as well as http payload.

• binary stream - Binary streams can be parsed based on binary decoding parameters that defines the bit pattern. A specific format to translate the bit stream to a json structure with name value pairs is defined for the user so they can query the data easily.

• http payload - HTTP payload will be parsed and queried based on the content-type of the packet.

Polling data from an outside resource

The user is able to poll and collect data from an outside resource by issuing periodic HTTP GET commands.

CANBUS OBD protocol driver with ELM327

The OBD driver is able to communicate with an ELM327 based OBD module using AT commands. The drive is able to extract the basic set of PIDs(sensor IDs) and relay information with the DMo engine at runtime.

Send user specified data upon event detection

When a JSON rule’s criteria is matched, DMo is now able to send user defined payload to the endpoint instead of the original payload that matched the event. This feature results in little to no modification on users’ end using their tools that can process specific form of data. Such an expected payload should be included in the "action.payload" section of the JSON rule. Users can request for matched packet’s network parameters like source and destination IP addresses, ports and protocol in their payload. The requested payload can include variable names from the original payload and the resulting payload will replace them with their corresponding values. Sample rules and examples are explained below.

Send Netflow

DMO is able to send netflow traffic data defined by the Cisco Netflow Version 9 Standard. The user is able to define network filters that will isolate traffic flow based on L3 network fields such as source/destination IP and source/destination ports. These filtered packets are consolidated and converted into a Cisco Netflow Version 9 netflow packet containing the stream information along with the netflow template. These packets are sent to a user defined collector at a specific endpoint to be processed.

Data in Motion Concepts and Terminology

The following sections provide information about Data in Motion concepts and terminology.

D3 structure overview

Dynamic Data Definition

Cisco Data in Motion's main configurable concepts and terminology is to support a Dynamic Data Definition (D3).

A D3 is a model representation for the underlying data present at the network. It consists of the following parts:

• Pattern—A pattern pertains to a combination of Protocol Parameters and Content Parameters.
• Condition—A condition refers to matching a pattern specifically defined within the request payload, such as the JSON body content.
• Action—An action consists of either a data management transaction relevant to the underlying data/D3 or a nested set of actions/D3s.

D3 takes the form of the following Condition(s)-Action(s) configurable rule syntax: IF Protocol Parameters AND Content Parameters THEN Actions

Where:

• Protocol and Protocol Parameters—Pertains to a combination of LINK protocol, NETWORK protocol, TRANSPORT protocol and APPLICATION protocol parameters. We have a PROTOCOL representation that includes both APPLICATION and NETWORK parameters; the CONTENT representation refers to the APPLICATION payload. In another case, if the PROTOCOL representation refers to 'ONLY' TRANSPORT protocol parameters (e.g. destination port), the CONTENT representation refers to the TRANSPORT payload.

• Content and Content Parameters—Refers to the payload, specifically, the data that is not part of any protocol header. This refers to the output of operations on the data sets that reside in the payload. This operation is based on natural parsing, decompression, data extraction, data summarization, decoding, data transformation, etc. The output inherits dynamically the underlying data models. For example, the metadata from pair values are naturally extracted as parametric metadata of any stream, or data records are naturally extracted and exposed from an XML, CSV or JSON payload into their underlying data model.
• Action—Could be one of the following: (action is dependent on Condition):
  • Transferring the matched packet payload over the network to a specified endpoint
  • Request data over the network from a specified endpoint
  • There are two types of rules:
  • Event-driven —An operation that is executed in the event of pattern matches against the real data.
  • Timer driven —These are scheduled and can be executed periodically based on a timer.

These D3 are defined in contexts. A context is a collection of related Data Definitions that can be accessed by an authorized group of users. Each user of a context has a separate username and login password for authentication and authorization security purposes. Each context will have a global (equivalent of root or super-user) password.

All authorized users have the same access rights to the D3's of a context. A D3 under a context will belong to one unique group. If no user-specified "groupid" is given, the "groupid" defaults to the Context name.

Windowing

Windowing defines meta basic information about a D3 concept. The D3 must contain exactly one windowing Meta primitive.

• Id—A unique Identifier that will identify the D3. It is the responsibility of the user to make sure the Id is unique.
• Context—Defines the container within which the D3 must be installed.
• Timer—Defines how often the D3 ACTIONS will be activated (in seconds)
• Cache—Defines how often the D3 ACTIONS will be activated (in buffer size)

Id and Context are required in every D3. The user must make sure that the Id assigned to a new D3 is unique; otherwise, the installation of D3 will fail.

The user must have login credentials to install a D3 within a specific Context.

Timer and Cache are optional primitives. There can only be a Timer primitive or a Cache primitive.

Network Control

Defines network filters and binary packet decoding of traffic. A D3 may contain 0 or 1 Network primitives.

• FilterBy—A primitive that defines the protocol of interest
• Decode—This primitive must be used when a binary packet must be decoded using a model

The FilterBy primitive must always be used in a Network primitive.

The Decode primitive is used only when binary packets must be decoded.

Application Control

Defines the application filters and content queries as follows:

• Application—Defines application filters and queries on the content. A D3 may contain 0 or 1 Application primitives.
• FilterBy—A primitive that defines the application protocol of interest.

The FilterBy primitive must always be used in an Application primitive.

Content Query—Defines queries on the content. A D3 may contain 0 or 1 Content primitives.

• Query—This primitive must be used when a query must be run on the content of the payload of interest.

The Query primitive must always be used in a Content primitive.

Dynamic Data Stream

A Dynamic Data Stream (DDS) is a user-defined stream that transmits the output of a D3 to a user-defined endpoint. A DDS can be defined to be a raw TCP stream or a HTTP stream depending on the requirements of the user. The output of a D3 is pipelined to the endpoint on a single connection.

If configured to be a HTTP or HTTPS stream a DDS performs a HTTP or HTTPS PUT operation on the endpoint.

Endpoint - This primitive specifies the destination Uniform Resource Identifier (URI) to be used by the action.

Dynamic Data Collector

A Dynamic Data Collector (DDC) is a special action that can be used to collect information from a device by initiating a connection to it at an endpoint address specified by the user. A DDC can be configured to connect to the device either using a raw TCP connection or using a HTTP or HTTPS GET depending on the capability of the device.

Endpoint—This primitive specifies the source Uniform Resource Identifier (URI) to be used by the action.

Data in Motion Configuration using JSON Format

The following figure illustrates the components of a JSON body reflecting the parameters needed. These parameters are explained as follows:

The JSON message that describes a D3 contains the following top-level blocks:

• Meta primitive—This block describes the meta-data about the D3.

• Network primitive—This block describes the network parameters.

• Application primitive—This describes the application level fields of interest.

• Content primitive—This describes what queries are to be run on the content/payload.

• Action primitive—This describes what actions to run and what the triggers are.

Meta (*REQUIRED)

The meta block contains the following fields:

• "ruleid"—Name of the D3 and servers as a reference. The ruleid MUST match the JSON filename of the HTTP PUT message.
• "context"—Specifies the context under which the D3 runs. Context name MUST match the authorized one in the HTTP authorization header.
• "timer"—Specifies how frequently the data that has been acquired should be processed. This is specified in milliseconds. The timer value MUST be greater than 1000 (1 sec).
• "cache"—Similar to the "meta.timer" field, specifies how frequently (in terms of bytes seen) the data should be processed. The cache value MUST be greater than 128 (bytes).

Network

The network block describes the network parameters of interest and specifies which data is to be processed. This block contains the following fields:

• "protocol"—Specifies the transport protocol we are interested in. ONLY TCP and UDP are allowed here.

• "decode"—If payload is encapsulated in TCP or UDP transport without a L7 application header, the decode format string can be used to specify how the payload should be decoded. This allows the user to specify the data model of the payload and to write queries on such payload. It is typically used in situations where the data generator (e.g. sensor) sends a packet with binary payload that needs to be interpreted as integers, characters or other data types.

• "Filter-by"—This block specifies the set of 6-tuples (source MAC, source IP, source port, destination, MAC, destination IP, destination port) the user is interested in. This block contains the following fields in the specified formats.

  • "srcmac"— FF:FF:FF:FF:FF:FF[/prefix length] : The prefix length can be used to specify a range of MAC addresses just like the ip prefix. The prefix range is limited to 1-48. Please see example below for more details.
  • "srcaddr"—ipaddress[/prefix length] : The prefix length can be used to specify a range of IP addresses. Currently, a prefix length of at most 8 (256) addresses is supported. Only raw IP addresses are supported.
  • "srcport"—Specifies the set of port number ranges, as well as, comma-separated values are accepted. e.g. [3212,1214,2118-2144] Max number of ports supported is 200.
    
  • "dstmac" —The format is similar to that of "srcmac".
  • "dstaddr"—The format is similar to that of "srcaddr".
  • "dstport"—The format is similar to that of "srcport".

The format of the mac address conforms to the standards of six groups of two hexadecimal digits separated by hyphens/colons as well as three groups of four hexadecimal digits separated by dots. Mac addresses for multiple devices can be tracked within the same rule by using a bit mask similar to the ip subnet mask. For example: AB:CD:EF:FF:FF:FF is for exact MAC address matching, AB:CD:EF:12:34:00/40 is for matching the first 40 bits of the MAC address, AB-CD-EF-FF-FF-FF and ABCD.EFFF.FFFF are supported formats as well.

Application

"application" :
{
   "protocol" : "http",
   "Filter-by" :
      {
         "Content-Type:" : "application/xml",
         "Host:"         : "10.12.13.14",
      },
}

This blocks specifies which application-level protocol fields we want to filter by.

• "protocol"—Specifies the application or L7 protocol for e.g. HTTP.
• "Filter-by"—This block matches directly on name-value pairs in the L7 header, where the name is supplied by the user. For example, user may want to match on "Content-Type" of XML from a specific HOST contained in HTTP protocol. The HTTP header must match EXACTLY as the standard header including case. The max number of application filters is 16. Please see example.

Content

Example:
"query" : "pressure > 23"
"query" : "\\[pressure\\]"
"query" : "press*"

This block specifies what queries are to be run on the content/payload.

• "query"—Specifies the query to be run. For example, it can be used to find payload where the pressure is greater than a threshold. Special characters in the query needs to be escaped with double backslashes \\. Wild character matching * is supported at the end of the character string.

Action (*REQUIRED)

This important block specifies what actions are to be taken in response to the filters and policies set earlier in the D3.

• "type"—This specifies the type of the action. Actions can be of two types. A D3 can specify either a list of event-driven actions or a single timer-driven action.

  • Event-driven actions are executed in response to filters or patterns specified in the D3. These actions are triggered when traffic matches the D3 specifications. For example, a D3 may specify that when JSON traffic is seen encapsulated in HTTP, then the original Header and the Original Payload should be processed. Multiple event-driven actions can be chained together. These actions are triggered when the data/traffic is processed which happens at the frequency of "meta.timer" or "meta.cache". The three event rules allowed are:
    • GetHeader - Send original HTTP header back. This action can only be specified if the rule is set up to filter HTTP payload and not just L3 TCP/IP parameters.
    • GetPayload - Send original payload or custom payload back. In case there is a raw tcp stream and binary decoding is enabled, the decoded message is sent back instead of the binary stream. The custom payload that is expected back should be mentioned in the action.payload section.
    • Netflow - Send netflow packets to the endpoint. Netflow should be the only action event type in a rule. Instead of the captured packet's payload, netflow will send the information regarding the network stream in a Cisco Netflow V9 defined packet. The "meta.timer" field is the time in ms that DMo waits before sending out the aggregated netflow information. The default value of "meta.timer" is 5 seconds. "Meta.cache" is not supported in the netflow rule definition.
  • Timer-driven actions are built-in actions that are not triggered by D3 matches on traffic but are run periodically at intervals. This interval is specified by "action.period". For example, we may want to specify that every 1000 milliseconds (1 second), Data in Motion should go fetch data from sensors (i.e. run the action FETCHDATA periodically). Timer-driven actions cannot be chained together and there must be only one specified.
    • FetchData - is the only timer actions that is supported as of now. This action performs an HTTP GET on the endpoint.

The typical usage is to have a FETCHDATA rule to periodically pull data from sensors and another rule that will actually process the data. DMo will automatically pick up the response to the HTTP GET command. It is not mandatory to have a pair of rules always but this is the expected usage.

• “payload”- This is specified only for "getPayload" actions. The payload sent to the endpoint should be specified here. Criteria for specifying the payload is as follows:

  • The length of the payload should be less than 1024 characters.
  • If the user needs value of a variable in the resulting payload, the variable should be included in content-query.
  • Variables should be enclosed in ‘#’, for example #variableName#.
  • Quotes should be preceded by two back slashes.
  • If the user wants a hash in the resulting payload, then it should be escaped by a hash. For example, if the user wants to see #var, then action.payload should be ##var.
  • If the user wants a quote in the resulting payload, then it should be preceded by two backslashes. For example, if the user wants to see "var", then action.payload should be \"var\"

• "period"—This is specified only for timer-driven actions and is ignored for event-driven actions other than netflow. It specifies the frequency of operation.

• "endpoint"—This block describes the endpoint URI for where the results of the actions are to be sent back in case of event rules and where to get the data in case of timer rules.

• "method"—Specifies the protocol to be used, there are four allowed protocols: http, https, tcp, and fifo.

  • http/https - the results will be encapsulated in an HTTP/HTTPS header with meta-data about the results.
  • tcp - the results will be send in a tcp stream.
  • fifo - a FIFO queue can be set up to store the results on the localmachine and is accessible through the listening port specified in the configuration file.

• "addr"—IP address of the destination endpoint.

• "port"—Port number for the destination endpoint. The port range MUST be in the range 1-65535.

• "resource"—Resource on the destination endpoint which will process the results.

The strings: method://addr:port/resource forms the full URI. Once a D3 has been registered, the client should listen at the specified URI for results.

Primitives for Decoding Raw Data

"network": {
                .
                .
                .
        "decode": DATA-MODEL
    }

The JSON payload syntax for decoding raw data is as shown.

The DATA-MODEL represents the binary data that will be decoded. If the raw data requires decoding, the decoded message along with the original binary stream will be sent out as hex values in a json format as the payload of the action block.

Data Models

The data model is used to interpret the binary data as basic data types. As of version 1.1, Cisco Data in Motion will support the interpretation of the following basic data types, such as char, uint16_t, uint32_t. The user has the option to associate variable names to each one of the interpreted fields and later perform queries using these variable names.

A data model is represented in the following format - {type, #of elements of this type, {var1, var2, var3, var4 …}}

• type—Defines the basic data type. 1 = char, 2 = uint16_t, 4 = uint32_t
• #of elements—Number of consecutive fields of this type that occur in the binary data
• {var1, var2, var3 … }—Name list field. It is a comma separated list of variable names that need to be associated with the binary fields. This is an optional field.

When it is present, the number of fields it contains must be equal to the # of elements. A user may choose to leave a variable name (field) blank if he does not want to associate a name with the binary field. A user may also choose to completely leave out the name list if there is no need to associate names with the variables.

Sensor Example

A sensor sends out temperature, pressure and humidity data as a 3byte payload. 1st byte corresponds to temperature, 2nd byte corresponds to the pressure and 3rd byte corresponds to humidity. In this situation the data model would be as follows

{1, 3, {t, pres, hum}}

This indicates that there are 3 consecutive bytes that need to be interpreted as u_char. The first byte will be associated the variable name 't', the second byte with variable name 'pres' and the third with 'hum'.

It is not necessary that all fields must be associated with a variable. For example, if the user chooses to associate only the first byte with a variable name on which he can write a query, the data model could be

{1, 3, {t,,}}

Notice that there must be "#of elements -1" commas in the variable name list field. But if a variable name is left blank then that field is not tracked. In another case, if the user wanted to track only the humidity field, the data model could be written as

1, 3, {,,hum}}

Binary Payload with Multiple Data Types Example

Binary payload with multiple data types are also possible. For example if the payload consists of a total of 10 bytes with the first 4 bytes to be interpreted as uint32_t, the next 4 bytes to be interpreted as 2 consecutive uint16_t with names 'vara' and 'varb' and the next 2 bytes to be interpreted as u_char with names 'ca' and 'cb' the data model would be as follows -

{4,1,{},2,2,{vara, varb},1,2,{ca,cb}}

Note that, we did not need to associate a name with the uint32_t field so the name list was left blank.

Skipping Bytes Example

In certain cases it may be required to skip some bytes as they may not hold any meaning in a particular context. For example, in a payload with 25 bytes, the first 21 bytes have no meaning and the last 4 bytes are to be interpreted as a uint32_t and associated with variable name 'h'. The data model would be

{0,21,4,1{h}}

Note that there is no “variable names” field following “0,21,”, as the 0 type does not take any variable names. In addition, empty braces (as shown in Example 14) must not be used. Note There will be strict checks on the format of the JSON and any failure in parsing will result in a HTTP 400 bad request at the time of installing of the rule. However, the correctness of the data model is the responsibility of the user and there will be no logical checks performed.

Primitives for Applying a Content-Type to Raw TCP/UDP Data

"network": {
                .
                .
                .
        "content": MIME_TYPE
    }

The JSON payload syntax for applying a content type to raw TCP/UDP payloads is shown.

The MIME_TYPE is a string that is one of the following supported MIME types –

1. application/json

2. text/xml

3. text/csv

4. text/html

5. text/plain

This primitive is useful when messages of one of the above formats are sent as a raw TCP/UDP payloads. By identifying the appropriate content type in the rule the user can help DMo invoke the appropriate parser for the payload.

By selecting an appropriate mime type the user can then apply a content query on the payload, as detailed in the following section, to improve the granularity of the event.

Primitives for Content-Query Condition

"content": {
        "query": QUERY-CONDITION
     }

The JSON payload syntax for a content-based query condition is shown.

A "Query Condition" is a condition set against the specific fields that are found in the payload. Since the fields are present in the real time in the data stream, the notion of real-time schema becomes a valid concept. The schema is derived from the data itself and thus not predetermined.

To determine the underlying schema, specific drivers are used to parse the data and extract, if possibly, an underlying schema.

ASCII Streams

< pcv:Descriptor rdf:about='org:1145'>
                    <pcv:label>Test and Evaluation Department</pcv:label>   
                  </ pcv:Descriptor>

The supported formats in the current release are CSV, XML, JSON, TXT, and HTML. * CSV format—A Comma Separated Values (CSV) does not specify any semantics about the values. The natural parsing of a CSV is that each column is of the same type. Thus, the parser will implicitly assign the letter A for the first column, B for the second column, and so forth, similar to how a Microsoft Excel spreadsheet represents its default column naming convention. In such an arrangement we can derive implicit attribute names, such as "A=value" or "AA= value", as shown below.

• XML format—The XML format represents structured and unstructured data. The structured data is represented in the attributes field in the XML tag. The unstructured data are found between begin and end tags as the values.

The schema representation of an attribute is the attribute itself. In the example below, rdf:about='org:1145' is the value-pair that will be used. Test and Evaluation Department will be used as free text.

• HTML and TXT formats—HTML and TXT formats generate free-text or unstructured data. Each extracted word from HTML and free text format is a valid keyword. The driver parser for HTML will attempt to discard HTML tags.
• JSON format—Key-value pair entry. The schema is inherited dynamically. For nested JSON object, the schema nomenclature follows a typical object model such as in "obj1.obj2.obj3=value."

Binary Streams

Binary requires the schema conversion be defined using the binary decoding string. Please prefer to Primitives for Decoding Raw Data section above.

Numeric Notations

JSON format example

{
   "Sensor":"GPS",
   "Timestamp":1326369894582,
      "State":  
       {
       "Longitude":71.30362551,
       "Altitude":25.5,
       "Latitude":42.66195771
       }
}

Query for GPS Pair-wise Values

"content":  {
   "query": "Sensor=GPS"
}

Query for a Specific GPS Altitude State Value

"content":  {
   "query":  "State.Altitude=25.5"
}

The system dynamically inherits numeric representation regardless of integer, or floating point representations or data resolution (e.g. 32bits or 64bits).

The "Query Condition" statement is the actual and effective representation of the key-value pair representation. For example, if the data or payload consisted of JSON format, then the object model is used. For example, if the data is given as shown in JSON format, then the examples of Data in Motion content query strings in JSON formats follow.

Query Operators

Query is a Simple String

"content": {
"query": "Temperature"
}

Query is a Simple String but Needs a Wild Character

"content": {
"query": "Temp*"
}

Query is an Equality Operation

"content": {
"query": "Temperature = 10.2"
}

Query is a Non-equality Greater-than Operation

"content": {
"query": "Temperature > 10"
}

Query is a Non-equality Less-than Operation

"content": {
"query": "Temperature < 10"
}

Query is a Range Operation

"content": {
"query": "Temperature =[10,20]"
}

The table describes the query operators, followed by examples.

Expression	Meaning
Free Text	Key words used as required constant
Text before and after =, <, >	Equality and non-equality operations
Text between =[ ]	Parameter for range evaluation

Query Algebra

with an "and" operation

"content": {
"query": "Temperature = 10.2F and Pressure = 20PSI"
}

with an "or" operation

"content": {
"query": "Temperature = 10.2F or Pressure = 20PSI"
}

The notion of algebraic operations is to achieve an operation on the data sets in real-time. The operators allowed are the following "or", "and" and "not". By default, the operator is an "and."

Query Escape Character

with an \\ to escape special characters

"content": {
"query": "Temperature = \\[10.2F\\]"
}

Certain characters are not allowed in the query text. Use the escape character double backslashes \\ in order to include special characters in the query text.

RESTful API

The following sections provide information about using the RESTful API.

Data in Motion Configuration uses RESTful service API implemented using HTTP methods (e.g. GET, PUT, POST, DELETE) and REST principles (e.g. hypertext driven URI). It also leverages from the Web-based Distributed Authoring and Versioning (WebDAV) set of extensions to the HTTP protocol to allow users to manage web resources (e.g. content, documents, videos, static, etc.) intuitively for collaborative editing and managing files on remote web servers. WebDAV is part of the Internet Engineering Task Force (IETF) approved protocol standard RFC 2518 and the latest proposed 4918 (See http://www.webdav.org/specs/ for additional information).

The API is designed to be RESTful with a JSON over HTTP data format. It supports four major categories of operations to manage D3 entries, such as Create, Read, Update, and Delete (CRUD): * Create a new D3 in a Context * Read by listing all existing D3 * Update an existing D3 in a Context * Delete an existing D3

Data in Motion RESTful-enabled HTTP server supports the WebDAV (RFC 2518 specifications) methods and commands that the “WebDAV Methods” section describes.

The RESTful APi can be used to install license and create context and install JSON rules.

Install License

You should have received a Cisco license key with your product. The license file must be located under the /home/localadmin/DMRoot/lic directory for Data in Motion services to be enabled. Optionally, the license file can be installed by using the WebDAV HTTP PUT command to upload to the Data in Motion enabled device. The admin user can upload/update using WebDAV HTTP PUT command and the HTTP header “Content-Type” MUST be set as “application/lic”.

Context Management

A context is an associated collection of related Data Definitions that can be accessed by an authorized set of users. Each user of a context has a separate login username and password for security authentication and authorization. Data in Motion offers RESTful APIs for an admin to create/modify/disable a context.

The admin must perform an HTTP POST with the corresponding URL patterns below and must also include the basic authorization header with username and password set to the admin user. Create context allows the DMo Administration to add a context user with a global password and configuration. Update context allows the Dmo Administrator to update username and password within a given context based on existing and new password. Disable context allows the DMo Administrator to disable the context and prevent users from logging in. You may be able to recover the contents of the context after this operation. (Request guidance from Cisco Support)

The URL pattern for create/modify/disable are formatted in the following ways:

Create Context https://<ipaddr>:<configport>/*/C/<context name>/<context_password>

Update Context https://<ipaddr>:<configport>/*/U/<context_name>/<new _password>

Disable Context https://<ipaddr>:<configport>/*/D/<context_name>

Where:

<ipaddr> - The IP of the host on which DMo is running

<configport> - Configuration port of DMo. Default is 443. Consult your admin if a custom port has been configured

<context_name> - Name of the context you are creating/updating

<context_password> - Password of the context you are creating

<new_password> - Must be different from the current password

An existing non-admin user with a valid password can use this API to modify their passwords. It cannot be used to create/disable a context.

To update the password on their own context, the user must perform a POST operation with the following URL pattern, and must include the basic authorization header with username and password set to the context name and current password, respectively

The URL pattern to update their own password on a context is:

Update context https://<ipaddr>:<configport>/*/U/<new_password>

Rule installation

The above mentioned D3 JSON rule can be installed using a HTTP PUT operation on the DMo server. The user must perform a PUT operation with the following URL pattern and must include the basic authorization header with usrname and password set to the context name and current password, respectively. The Content-Type of the HTTP PUT message MUST be set to "application/config" and the filename MUST end in a .json extension.

Rules can be enabled or disable during processing. A disabled rule will not execute the actions defined in the D3 structure.

Rule Creation - https://<ipaddr>/<rulename>.json using HTTP PUT or POST

Rule Deletion - https://<ipaddr>/<rulename>.json using HTTP DELETE

Rule Disable - https://<ipaddr>/<rulename>.json/D using HTTP PUT or POST

Rule Enable - https://<ipaddr>/<rulename>.json/E using HTTP PUT or POST

To update a rule, simply send a HTTP PUT message with the same rule name and the original rule will be overwritten.

WebDAV Methods and Example

The following sections describe the WebDAV methods:

PROPFIND

PROPFIND Client Request (e.g. Retrieves all properties)

PROPFIND / HTTP/1.1
Host: www.cisco.com
Accept-Language: en-us
Authorization: Basic cm9vdDpyb290(NOTE: Basic Authentication with Base64 username:password encoding)
Connection: Keep-Alive
Content-Language: en-us
Content-Length: xxx
Content-Type: text/xml
Depth: 0
Translate: f
User-Agent: Microsoft Data Access Internet Publishing Provider DAV

<?xml version="1.0" encoding="utf-8" ?>
   <D:propfind xmlns:D="DAV:">
     <D:allprop/>
   </D:propfind>

PROPFIND Server Response (e.g. Retrieves all properties)

HTTP/1.1 207 Multi-Status
Date: Mon, 10 Sep 2007 22:08:38 PDT
Server: DM Version 1.0(Unix)
DAV: 2
MS-Author-Via: DAV
Stream-Length: -1
Content-Type: text/xml; charset=ISO-8859-1
Connection: close

<?xml version="1.0" encoding="utf-8" ?>
<D:multistatus xmlns:D="DAV:">
<D:response>
<D:href>https://www.cisco.com/Definitions.json</D:href>
<D:propstat>
            <D:prop>
            <D:creationdate></D:creationdate>
            <D:getlastmodified>Sat, 21 Jul 20013 PDT</D:getlastmodified>
            <D:resourcetype>
                <D:collection></D:collection>
</D:resourcetype>
            <D:contenttype></D:contenttype>
                                    <D:getcontentlength>4096</D:getcontentlength></D:prop>
            <D:status>HTTP/1.1 200 OK</D:status>
        </D:propstat>
    </D:response>
</D:multistatus>

Retrieves the properties of a D3 (e.g. href, author, creationdate, displayname, etc.). PROPFIND returns a multi-status server-side response with XML body content as shown in these examples.

PUT

PUT Client Request

PUT /myContext.json HTTP/1.1
Host: www.cisco.com
Accept-Language: en-us
Authorization: Basic cm9vdDpyb290
Connection: Keep-Alive
Content-Language: en-us
Content-Length: 123
Content-Type: application/config
User-Agent: Microsoft Data Access Internet Publishing Provider DAV

PUT Server Response

HTTP/1.1 201 Created
Date: Mon, 10 Sep 2007 22:08:38 PDT
Server: DM Version 1.0(Unix)
Content-Type: text/xml; charset=ISO-8859-1
Connection: close

Upload a WebDAV resource, such as a document or image through request URI.

DELETE

DELETE Client Request

DELETE /myContext.json HTTP/1.1
Content-Type: application/config
Authorization: Basic cm9vdDpyb290
Accept: */*
Host: www.cisco.com

DELETE Server Response

HTTP/1.1 204 No Content
Date: Mon, 10 Sep 2007 22:08:38 PDT
Server: DM Version 1.0(Unix)
Connection: close

Removes a resource (e.g. collection or file).

GET

GET Client Request

GET /favicon.ico HTTP/1.1
Authorization: Basic cm9vdDpyb290
Host: www.cisco.com
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 1.0.3705; .NET CLR 2.0.50727)

GET Server Response

HTTP/1.1 200 OK
Date: Mon, 10 Sep 2007 22:08:38 PDT
Server: DM Version 1.0(Unix)
DAV: 2
MS-Author-Via: DAV
Stream-Length: 3064
Content-Type: image/x-icon; charset=ISO-8859-1
Connection: close

Retrieves or download a resource or HTTP resources through Request URI

Sample RESTful API using JSON Data Format Use Cases

PUT /getData.json  HTTP/1.1
Host: 128.107.165.80 : 8082
Authorization: Basic T3BlbkNvbnRleHQ6Zm9vYmFy
Content-Type: application/config
Content-Length: 727

{
          "meta ":  {
                "ruleid" : "getData",
                "context" : "OpenContext",
                "timer" : "3000"
                },
       "network ":  {
                "protocol" : "tcp" ,
                 "Filter-by" : {
                           "srcaddr" : "172.27.231.28",
                           "dstaddr" : "10.1.1.2",
                             "sport" : "80",
                             "dport" : "4002",
                          },
                },
    "application    ":  {
                "protocol" : "http" ,
                "Filter-by" : {
                        "Content-Type" : "application/json" ,
                          },
                },
        "action ":  {
                "type" : "event",
                "name":  [ "GetPayload" ] ,

                "endpoint" :  {
                        "method" : "http" ,
                        "addr" : "172.27.231.28" ,
                        "port" : "5001" ,
                        "resource" : "/process/sensor.pl" ,
                          },
                },
}

PUT /vijay12.json  HTTP/1.1
Host: 128.107.165.80 : 8082
Authorization: Basic T3BlbkNvbnRleHQ6Zm9vYmFy
Content-Type: application/config
Content-Length: 845

{
          "meta ":  {
                "ruleid" : "vijay12",
                "context" : "OpenContext",
                "timer" : "3000" ,
                },
       "network ":  {
                "protocol" : "tcp" ,
                 "Filter-by" : {
                           "srcaddr" : "172.27.231.28",
                           "dstaddr" : "10.1.1.2",
                             "sport" : "80",
                             "dport" : "4002",
                          },
                    "decode" : "(2,1,{A}, 4,2,{B,})",
                },
                "content" :  {
                             "query" : "A>23" ,
                         },
                },
        "action ":  {
                "type" : "event",
                "name":  [ "GetPayload" ] ,
                "endpoint" :  {
                        "method" : "http" ,
                        "addr" : "172.27.231.28" ,
                        "port" : "5001" ,
                          },
                },
}

PUT /vi12.json  HTTP/1.1
Host: 128.107.165.80 : 8082
Authorization: Basic T3BlbkNvbnRleHQ6Zm9vYmFy
Content-Type: application/config
Content-Length: 393

{
          "meta ":  {
                "ruleid" : "vi12",
                "context" : "OpenContext",
                },
        "action ":  {
                "type" : "timer",
                "name":  [ "Fetchdata " ] ,
                "period" : "10",
                "endpoint" :  {
                        "method" : "http" ,
                        "addr" : "172.27.231.28" ,
                        "port" : "5001" ,
                        "resource" : "/process/sensor.pl" ,
                          },
                },
}

Examples are shown.

Data in Motion Command Line

The centOS OVA installs the Data in Motion binary executables at the following locations: * /usr/sbin/dm - Data in Motion software

Synopsis: <-i en1> -d (packet dump) -c -b (rebuild rules)

-i—Allows the DMo Administrator (e.g. run as root user or as sudo root) to define the network interfaces or devices to bind to with a maximum of 5 (e.g. -i lo -i eth1 -i eth2 -i eth3 -i eth4).

-d—Optionally, allows the DMo Administrator to print the packet-level dump information that is useful for verbose and debugging purposes.

-c—Optionally, allows the DMo Administrator to set the configuration directory containing one or more configuration files. Default configuration directory is set to "/home/localadmin/DMRoot/cfg/".

-b—Optionally, allows the DMo Administrator to rebuild all the rules that is referenced under the configuration directory structure.

Data in Motion Configuration

<_filter_url_>
  FitlerURL
</_filter_url_>

<_mem_prefix_>
  /home/localadmin/DMRoot/mem
</_mem_prefix_>

<_ctx_prefix_>
  /home/localadmin/DMRoot/contexts
</_ctx_prefix_>  

<_log_prefix_>
  /home/localadmin/DMRoot/log
</_log_prefix_>  

<_crt_prefix_>
  /home/localadmin/DMRoot/cert/dm.crt
</_crt_prefix_>

<_key_prefix_>
  /home/localadmin/DMRoot/cert/dm.key
</_key_prefix_>

<_adm_prefix_>
  /home/localadmin/DMRoot/lic
</_adm_prefix_>

<_ip_network_>
  127.0.0.1
</_ip_network_>

<_listenPort_>
  8443
</_listenPort_>

<_configPort_>
  443
</_configPort_>

The default Data in Motion configuration file uses an XML-based format constructs with the following definitions:

• mem_prefix—Sets the path to the data streams.
  
• dci_filter—Sets the packets filter type (e.g. UDP, TCMP, ICMP)
  
• ctx_prefix—Sets the default absolute path name for root context. No chdir occurs for path.
  
• log_prefix—Sets the default absolute pathname for log directory.
• adm_prefix—Sets the default admin prefix for Cisco FLEXlm license keys.
• filter_url—Sets the default rule name for URL filter. Must sets context and schema.
• ip_network—Sets the IP address of node to connect with as the node synchronization.
• MAXtimeout—Sets the client timeout in floating point seconds.
  
• listenPort—Sets the port number to listen from.
  
• configPort—Sets the port number to configure from.
  
• auth_basic—Sets the Base64 encoding of pair value token HTTP Basic Authentication header.
• crt_prefix—Sets the path location to X.509-compliant SSL certificate.
  
• key_prefix—Sets the path location to X.509-compliant SSL key.

The syntax of a sample Data in Motion configuration file is shown, which is located by default at /home/localadmin/DMRoot/cfg/dm.cfg

OBD protocol driver

obd.conf file

{
    "obd-connect-serial" : "/dev/ttyUSB0",
    "obd-connect-serial-baud" : "38400",
    "obd-listen-parent-interface" : "enp0s25",
    "obd-listen-network-prefix" : "10.0.0.0",
    "obd-listen-network-prefix-length" : "16",
    "obd-listen-vlan-tag" : "100",
    "obd-cache-expire-timeout-ms" : "5000"
}

obd.cfg file

<_CGI_BINARY_>
/usr/sbin/obd:5000
</_CGI_BINARY_>

The CANBUS protocol is widely used in the transportation industry. It is used to interact with the ECUs in cars and trucks. The DMo in this case interacts with the ELM327 which is a programmed microcontroller that translates low level CANBUS protocol into a simple set of interfaces that can be called via UART. By connecting with the ELM327 microcontroller, the OBD driver is able to communicate between the sensors on the car/truck using simple AT commands. This is through either directly using serial port or via TCP encapsulated connection.

• Serial port connection the OBD module opens a serial device (exmaple: /dev/ttyUSB0) where an ELM-compatible adapter is connected. The obd-connect-serial option is set in the obd.conf file. I

• Encapsulated TCP conneciton when the serial device is not directly available, TCP can be used to encapsulate and forward the data between the OBD DMo module and the ELM327. In this case, OBD opens a TCP socket, binds it to the specified address and port and listens for a TCP conneciton. the obd-connect-tcp option is set in the obd.conf file.

• Sensor virtualization OBD module creates a VLAN-tagged subinterface(called "obd") on a configured parent interface. When the OBD PID discover is complete, the corresponding IP addresses are created ont eh sub-interface and OBD module listens on these IP addresses for HTTP requests fetching the OBD data. For each discovered obdmode/obd-PID combination, the corresponding IP address is created in teh form of abc.def... where abc.def is derive dfrom the configured netowrk prefix. For example, if the netowrk prefix is configured to be 10.0.0.0 with prefix length being 8, then for mode 1, PID 14, the IP address is 10.0.1.14. Consequently, a HTTP GET request on http://10.0.1.14 would fetch the OBD drata for mode 01, PId 14.

In order to use the OBD protocol driver, two configuration files are need, obd.cfg and obd.conf. These files should be included in the DMRoot/cfg/ directory. Please refer to the following OBD example for more information.

• obd.cfg The obd.cfg starts the OBD module during DMo startup. It points to the obd module binary and listens at the port that is provided in the configuration. In the example shown, the obd module is listening on port 5000.

• obd.conf The obd conf file shows general obd module configuration

  • obd-connect-tcp - TCP listen string in the form "host:port"
  • obd-connect-serial - serial device to open from
  • obd-connect-serial-baud - baud rate to use when using the serial port
  • obd-listen-parent-interface - parent interface to use when creating obd interface. The interface must exist when starting DMo.
  • obd-listen-network-prefix - network prefix to use when creating the IP addresses for OBD PIDs. The network prefix MUST be an IPv4 network.
  • obd-listen-netowrk-prefix-length - length of the netowrk prefix, MUST be <= 16.
  • obd-listen-vlan-tag - VLAN tag associated with the sub-interface. The VLAN tag MUST be in the range of <0, 65535>
  • obd-cache-expire-timeout-ms - the duration for obd to cache the PID values

Send a custom payload upon event detection

Users have web applications that can accept data in certain formats. Data in Motion can readily be used with such applications, by sending a customized payload that users want instead of sending the entire original payload that matched the event. This payload can be mentioned in the rule as part of the action section.

Users can also request for network parameters of the packet that matched the event. The network parameters that can be requested in the payload are protocol, source IP address, destination IP address, source port, destination port, source MAC address, destination MAC address and payload. These parameters are recognized by the below keywords enclosed in hashes.

* dmo_protocol

* dmo_srcip

* dmo_dstip

* dmo_srcport

* dmo_dstport

* dmo_srcmac

* dmo_dstmac

* dmo_payload

A sample custom payload rule looks like this.

Sample Custom Payload rule

{

        "meta"  :       {
                                "ruleid" : "tcpsend",
                                "context" : "test",
                                "cache" : "129",
                                },

        "network":      {
                                "protocol" : “http”,
                                 "filter-by": {
                                        "dport" : "5005",
                                 },
                        },

        "content"       :       {
                                        "query" : “cool=199”,
                                },

        "action"        :       {
                                "type" : "event",
                                "name" : [ "getpayload" ] ,
                                "payload" : "This is an ##example. IP Address: #dmo_srcip#, Matched value: #cool#”,
                                "endpoint" :  {
                                                "method" : "http" ,
                                                "addr" : "10.41.56.41" ,
                                                "port" : "5559"

                                              }
                                }
}

This is an event rule that tracks http data to port 5005 and also queries each payload to check if “cool = 199”. If this query matches, the custom payload mentioned in action.payload is sent out. Note that the custom payload is requesting for #dmo_srcip#, the IP address from which the matched packet was received and the matched value #cool#. Notice that there are two hashes before the word example. If users want to see a hash in their payload, they should prepend that hash by another hash.

Payload received at port 5005

POST /json1 HTTP/1.1
Content-Type: application/json
User-Agent: Java/1.8.0_31
Host: 1.100.50.27:5005
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 259

{“cool”:”199”,”heat":"250","atmp":"2","mpg":"16","imp":"30","cer":"NODATA","itmp":"37 F","maf":"16.57 g/s","ta":"1.2 deg","rpm": 0,"tpos":"21 %","ft":"-10 %","spd":"22 mph","bar":"NODATA","ampg":"29.9 mpg","fp":"NODATA"}}

This payload matched the content query and the action is to send the custom payload to the endpoint 10.41.56.41 at port 5559.

Output sent to port 10.41.46.41:5559

PUT / HTTP/1.1
Connection: Keep-Alive
User-Agent: DM-1.0
Data-Stream: /centos/10.41.56.41-1.100.50.27--0-5005--0+1
Date: Thu, 03 Nov 2016 10:35:19 GMT
Content-Type: application/octet-stream
Host: 10.41.56.41
Content-Length: 66

This is an #example. IP Address: 10.41.56.41, Matched value: 199

Send Netflow data

The following section describes the netflow use case by DMo. Please refer to Cisco NetFlow technology Version 9 for more detailed information [http://www.cisco.com/en/US/technologies/tk648/tk362/technologies_white_paper09186a00800a3db9.html].

Netflow template used by DMo

        FlowSet Id: Data Template (V9) (0)
        FlowSet Length: 36
        Template (Id = 273, Count = 7)
            Template Id: 273
            Field Count: 7
            Field (1/7): IP_PROTOCOL_VERSION
                Type: IP_PROTOCOL_VERSION (60)
                Length: 1
            Field (2/7): PROTOCOL
                Type: PROTOCOL (4)
                Length: 1
            Field (3/7): L4_SRC_PORT
                Type: L4_SRC_PORT (7)
                Length: 2
            Field (4/7): IP_SRC_ADDR
                Type: IP_SRC_ADDR (8)
                Length: 4
            Field (5/7): L4_DST_PORT
                Type: L4_DST_PORT (11)
                Length: 2
            Field (6/7): IP_DST_ADDR
                Type: IP_DST_ADDR (12)
                Length: 4
            Field (7/7): TCP_FLAGS
                Type: TCP_FLAGS (6)
                Length: 1

Netflow is used to monitor traffic flow defined by the user. DMo enables the user to collect netflow records extracted from the captured traffic flow. These flows are sent to a specific endpoint to be collected and processed.

A sample netflow rule would contain the following:

Sample netflow rule

{
        "meta": {
                "ruleid": "tcpNetflow",
                "context": "test",
                "timer" : "5000",
        },
        "network": {
                "protocol": "tcp",
                "filter-by": {
                  "srcaddr" : "171.70.55.171",
                  "dport" : "5005",
        },
        },
        "action": {
                "type" : "event",
                "name": ["netflow"],
                "endpoint": {
                  "method": "tcp",
                  "addr": "171.70.55.171",
                  "port": "5559"
                },
        },
}

This rule will filter traffic that originates from IP address 171.70.55.171, sent to port 5005 and collect the information defined by the DMo netflow template. Dmo will send out the aggregated netflow record every 5 seconds as defined by the meta.timer. This netflow record will be sent to the endpoint http://171.70.55.171:5559/ in a tcp stream.

Sample netflow record

Cisco NetFlow/IPFIX
    Version: 9
    Count: 2
    SysUptime: 1459968000
    Timestamp: Nov 10, 2016 03:34:58.000000000 PST
    CurrentSecs: 1478777698
    FlowSequence: 0
    SourceId: 0
    FlowSet 1
        FlowSet Id: Data Template (V9) (0)
        FlowSet Length: 36
        Template (Id = 273, Count = 7)
            Template Id: 273
            Field Count: 7
            Field (1/7): IP_PROTOCOL_VERSION
                Type: IP_PROTOCOL_VERSION (60)
                Length: 1
            Field (2/7): PROTOCOL
                Type: PROTOCOL (4)
                Length: 1
            Field (3/7): L4_SRC_PORT
                Type: L4_SRC_PORT (7)
                Length: 2
            Field (4/7): IP_SRC_ADDR
                Type: IP_SRC_ADDR (8)
                Length: 4
            Field (5/7): L4_DST_PORT
                Type: L4_DST_PORT (11)
                Length: 2
            Field (6/7): IP_DST_ADDR
                Type: IP_DST_ADDR (12)
                Length: 4
            Field (7/7): TCP_FLAGS
                Type: TCP_FLAGS (6)
                Length: 1
    FlowSet 2
        FlowSet Id: (Data) (273)
        FlowSet Length: 109
        Flow 1
            IPVersion: 04
            Protocol: 6
            SrcPort: 52291
            SrcAddr: 171.70.55.171 (171.70.55.171)
            DstPort: 5005
            DstAddr: 1.100.50.50 (1.100.50.50)
            TCP Flags: 0x02
        Flow 2
            IPVersion: 04
            Protocol: 6
            SrcPort: 52291
            SrcAddr: 171.70.55.171 (171.70.55.171)
            DstPort: 5005
            DstAddr: 1.100.50.50 (1.100.50.50)
            TCP Flags: 0x18

Sensor Demo Examples

This section provides two examples. The first is of a simple event rule and the second is of an end-to-end scenario that covers the Data in Motion capabilities of virtualizing sensors in a car and being able to access the data from those sensors.

Example 1

The setup for this example consists of two VMs. One of the VMs runs a simple C application that sends data to an IP address and port of our choice and also listens on port 7766. We call this the Test VM. The other VM runs Data in Motion, which has the rule installed in it.

Event rule setup

This example installs a rule into Data in Motion to track tcp stream of data coming from the source address 192.168.56.102 to the port 7765. An action is performed on the stream of data that matches these criteria. For this rule, the action is to post the payload to the address 192.168.56.102 on to the port 7766. The rule installed looks like this:

Installed rule Installed-rulejson

The test VM sends UDP data to the DMo machine (192.168.56.102) on port 7765

Send TCP data

The DMo machine should be listening on the port 7765 since this is a TCP connection. This is not necessary for other types of connections.

Listen on port

Based on the event rule, DMo now sends the matched stream of data to the endpoint mentioned in the action.

Received data on Test VM

Example 2

OBD setup

This is a set up that consists of a car emulator that emulates the data from different sensors inside a car (OBD-II emulator). The emulator is connected to an ELM 327 interface that supports the OBD protocol and converts OBD data into serial output. The ELM 327 is then connected to the VM running Data in Motion on the other end through USB. Data in Motion has an OBD driver inside it that has the capability of reading the serial output from the ELM327 and converting into JSON readable format.

Each sensor inside a car is associated with a Parameter ID known as the PID. OBD-II PIDs are codes used to request data from a vehicle. The OBD driver virtualizes each sensor inside the car into an IP address that is mapped to its PID. DMo can now read the sensor data through these IP addresses.

Examples of JSON formats from different virtualized IP addresses for different sensors are as below. The curl command can be used to get the status of these PIDs.

curl 10.0.1.15
{"status":"ok","responses":[{"ecu_id":2024,"mode":1,"pid":5,"data":[114]},{"ecu_id":2025,"mode":1,"pid":5,"data":[114]}]}
curl 10.0.1.12
{"status":"ok","responses":[{"ecu_id":2024,"mode":1,"pid":12,"data":[149,169]},{"ecu_id":2025,"mode":1,"pid":12,"data":[149,169]}]}
curl 10.0.1.13
{"status":"ok","responses":[{"ecu_id":2024,"mode":1,"pid":13,"data":[131]},{"ecu_id":2025,"mode":1,"pid":13,"data":[131]},{"ecu_id":2026,"mode":1,"pid":13,"data":[131]}]}

Changes in configuration

There are two configuration files that should be changed for enabling the OBD driver module in Data in Motion. They are both located in /home/localadmin/DMRoot/cfg/ and are called obd.cfg and obd.conf. *The obd.cfg file consists of the path where the obd binary is located and the port at which the binary operates. The three lines in this config file should be commented. *The obd.conf file consists of two parameters that need to be changed: "obd-connect-serial" : "/dev/ttyUSB0" – This is the path to the USB port that is connected to the DMo machine from the ELM327. The name of this USB port can be obtained in the /dev directory.

TTYUSB0

"obd-listen-parent-interface":"enp0s3" – This is the interface on which data in motion is running. Changes to these two fields must be made accordingly. Restart DMo after these changes to see the obd up and running.

End-to-end scenario

The below example show the end-to-end scenario of how data can be retrieved from sensors in a car:

DMo with timer and event rules

Two rules are needed for this scenario. One is a timer rule that fetches data from an IP address every 10 seconds. In this case, this IP address is 10.0.1.13, which is the virtualized IP address of a sensor that resides in the car. 10.0.1.13 maps to the PID 13, which represents Vehicle Speed. This results in data that is generated from the IP 10.0.1.13.

OBD Timer rule The other rule is an event rule that matches data coming from the IP 10.0.1.13 and sends the matched payload to an endpoint 192.168.56.102, which is the test VM. Using netcat command on the test VM side, the data coming from the 10.0.1.13 IP can be verified.

OBD Event rule

Note that the OBD data is sent on port 80 by default. So, on the receiving end a network-filter may be added to match data coming from the corresponding PID IP on port 80.

The Test VM being the receiving endpoint waits for OBD data on port 9880 since the event rule drives the matched payload onto the test VM to port 9880.

Data received on endpoint

Acronyms and Terms

DDC—Dynamic Data Collection is a special action that can be used to collect information from a device by initiating a connection to it at the endpoint address specified by the user.

DDD (D3)—Dynamic Data Definition is a model representation of the underlying data present at the network.

DDS—Dynamic Data Stream is a user-defined data stream that transmits the output of a D3 to a user-defined endpoint.

DMo—Data in Motion reflects the instantiation of TigerMe in the scope of Cisco productization roadmaps.

JSON—JavaScript Object Notation used for standard data interchange format.

TigerMe—Name of a technology that Cisco acquired in 2012.

WebDAV—Web Distributed Authoring and Versioning is a set of standard HTTP extensions that allows users to manage web resources. It is part of the Internet Engineering Task Force (IETF) approved protocol standard RFC 2518.

XML—eXtensible Markup Language (XML) used for standard data interchange format markup language.