Latest changes as RSS feed.

With this update, the data streamer now includes more comprehensive Usage and Event data to the destination buckets, Kinesis streams and user-provided applications.

Additional S3 Data (CSV)

Note:
Users of the S3 integration should ensure that minimum security standards are enforced. Details of configuring buckets for this purpose are described in Security Guidelines.

Event Data

The following additional properties are now included in event stream CSV files as column headers (where applicable):

  • endpoint_id
  • endpoint_name
  • endpoint_imei
  • endpoint_ip_address
  • endpoint_tags
  • event_source_id
  • event_source_name
  • sim_id
  • sim_iccid
  • msisdn_msisdn
  • sim_production_date
  • imsi_id
  • imsi_imsi
  • user_id
  • user_username
  • user_name

Usage Data

The following additional properties are now included in usage stream CSV files as column headers (where applicable):

  • endpoint_name
  • endpoint_ip_address
  • endpoint_tags
  • endpoint_imei
  • msisdn_msisdn
  • sim_production_date
  • operator_mncs
  • country_mcc

Additional Kinesis Data (JSON)

Kinesis Event Data

The following additional properties are now included in event stream JSON for endpoint, sim and imsi objects:

{
 "endpoint" : {
   "name":  "...",
   "imei":  "...",
   "ip_address": "...",
   "tags": "..."
 },
 "sim" : {
   "msisdn": "...",
   "production_date": "..."
 },
 "imsi" : {
   "imsi": "...",
   "import_date": "..."
 }
}

Kinesis Usage Data

The following additional properties are now included in event stream JSON for endpoint, sim and operator objects:

{
  "endpoint" : {
    "name": "...",
    "ip_address": "...",
    "tags": "...",
    "imei": "...",
  },
  "sim": {
    "msisdn": "...",
    "production_date": "..."
  },
  "operator": {
    "mnc": "...",
    "country": {
      "mcc" : "..."
    }
  }
}

Additional Rest API Data (JSON)

The data streamer may also send usage and event data in JSON format toward a configurable URL specified by the user. In this case, users provide an application which consumes HTTP POST requests sent from the BICS platform. With this deployment, both Rest API stream types (Rest and Bulk) will include the additional data as specified in Kinesis Event Data and Kinesis Usage Data.

Security Guidelines

Event data that is sent via Data Streams may include usernames, email addresses and other data which can identify users or platform resources. The generated .csv files should therefore be treated as containing sensitive information. Precautions should be taken to ensure that the event and usage data in the destination S3 buckets are adequately secured.

The following three steps should be considered as the minimum security requirements for storing such data in S3:

  • Ensure that the S3 bucket is not publicly accessible. This can be applied in the Permissions tab of the S3 bucket.
  • Server-Side Encryption can be enabled per-bucket and S3 will encrypt objects before they are saved to disk. The decryption is then performed when downloading the objects. This can be enabled in the Properties tab of the S3 bucket.
  • The IAM user whose credentials (key ID & key secret) access the S3 bucket should have their permissions restricted to writing to the required bucket only. This can be done in the AWS console in Users -> {Data Stream User} -> Permissions -> Add Permissions -> Attach Policy Directly -> Create Policy. An example JSON Policy would look like:
{
  "Id": "Policy1569854716005",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1569854710254",
      "Action": [
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::my-data-stream", // <1>
      "Principal": {
        "AWS": [
          "data-stream-writer" // <2>
        ]
      }
    }
  ]
}

<1> ARN of the destination S3 Bucket
<2> The IAM user with data stream write access

TIP
AWS provide an online JSON Policy Generator which can be used to create a policy like the example given above.

Backwards Compatibility

The BICS Data Streamer is under active development and is updated for performance and quality improvements regularly so that users of the platform may gain rich streaming insights into usage and event data.

Versioning
There is no external versioning of the Data Streamer that is necessary for developers to have to track. Updates are, therefore, always performed on the service with the intent of preserving backwards compatibility. This means that existing JSON or CSV entities have their ordering preserved and are not renamed or replaced when updates to the data streamer are performed.

Parsing S3 or Kinesis Artifacts
Users who have built custom integrations in AWS or otherwise which consume the JSON or CSV generated from S3 or Kinesis streams should expect that additional JSON or CSV data may be added in future. Mature and tested libraries designed for parsing or reading CSV and JSON data should be used for custom integrations (which may be in lambdas, for example) to ensure compatibility in cases where additional properties or objects are added to data streams in future.

Endpoint Quota Changes

The following endpoints for managing endpoint quota have been improved.

Topic Method Entrypoint Description
Endpoint POST /api/v1/endpoint/{endpoint_id}/quota/data Set Data Quota
Endpoint GET /api/v1/endpoint/{endpoint_id}/quota/data Retrieve Data Quota details
Endpoint POST /api/v1/endpoint/{endpoint_id}/quota/sms Set SMS Quota
Endpoint GET /api/v1/endpoint/{endpoint_id}/quota/sms Show SMS Quota details

Traffic Limit Changes

Method Entrypoint Status
POST /traffic_limit Removed

Traffic Limit POST Details

The POST method for /traffic_limit has been marked deprecated and is now removed from the documentation.

Traffic Limits may still be applied by adding existing traffic limits to a service profile with the following operation:

PUT /api/v1/service_profile/{profile}/service/{service}/traffic_limit/{traffic_limit}

For more information on adding traffic limits to services, see the Swagger Reference for this entrypoint.

API Rate Limiting Details

The supplementary documentation adds details on newly-applied logic for API rate limiting.

The API has rate limiting enforced at a value of 2000 requests per IP in a five minute window.

When this threshold is reached, 403 Forbidden will be returned by the system as a response to excessive requests until the average request rate is within the specified limit.

This is documented and may be easily referenced in the API Conventions page.

RAT Type Changes

A new parameter has been added to the following RAT Type endpoint:

Topic Method Entrypoint Description
RAT Type GET /rat_type Returns a list of supported RAT types

QoS Definitions

A new API for defining QoS Bandwidth limits per PDP context definition and RAT Type has been added

Topic Method Entrypoint Description
PDP Context Definitions POST /pdp_context_definition/{pdp_id}/qos_definition Add a new QoS Limit
PDP Context Definitions GET /pdp_context_definition/{pdp_id}/qos_definition Get the limits for the given PDP context definition and RAT Type
PDP Context Definitions PATCH /pdp_context_definition/{pdp_id}/qos_definition Modifies the limits for the given PDP Context and RAT Type
PDP Context Definitions DELETE /pdp_context_definition/{pdp_id}/qos_definition Remove the limits

Furthermore, the PDP Context get definition call has been modified to include the qos definitions as well

Topic Method Entrypoint Description
PDP Context Definitions GET /pdp_context_definition Get the list of pdp context definitions

SMS Routing Changes

The following endpoints for configuration of SMS Routing entries have been updated:

Topic Method Entrypoint Description
SMS Routing GET /sms_routing Get list of SMS routing entries
SMS Routing GET /sms_routing/{sms_routing_id}/data Get list of SMS routing data
SMS Routing POST /sms_routing/{sms_routing_id}/data Create SMS routing data
SMS Routing PATCH /sms_routing/{sms_routing_id}/data/{sms_routing_data_id} Update SMS routing data

Data Streamer

The following entrypoint to delete Data Streams has been added:

Topic Method Entrypoint Description
Data Streamer DELETE /data_stream/{data_stream_id} Delete a Data Stream by ID

Data Streamer (1.2.14)

The following entrypoint to list and manage Data Streams has been added:

Topic Method Entrypoint Description
Data Streamer GET /data_stream List Data Streams
Data Streamer POST /data_stream Create Data Stream

Additional documentation pages have been added for the following Data Streamer integrations:

RAT Type Additions

The following endpoint to retrieve a list of RAT types has been added:

Topic Method Entrypoint Description
RAT Type GET /rat_type List RAT types

Network Coverage Data RAT Type Blacklist Additions

The following endpoints have been added for blacklisting RAT Types on network coverage data:

Topic Method Entrypoint Description
Network Coverage PUT /network_coverage/{coverage_id}/coverage/{operator_id}/rat_type/{rat_type_id} Blacklist RAT type
Network Coverage DELETE /network_coverage/{coverage_id}/coverage/{operator_id}/rat_type/{rat_type_id} Remove a RAT type from the blacklist

Network Coverage Data Additions

The following endpoint has been updated to show a list of blacklisted RAT types on a network coverage data entry:

Topic Method Entrypoint Description
Network Coverage GET /network_coverage/{coverage_id}/coverage Retrieve Network Coverage Data

SMS Routing Additions

The following endpoints for configuration of SMS Routing entries have been added:

Topic Method Entrypoint Description
SMS Routing GET /sms_routing Get list of SMS routing entries
SMS Routing POST /sms_routing Create a new SMS routing entry
SMS Routing GET /sms_routing/{sms_routing_id} Fetch SMS routing entry by ID
SMS Routing PATCH /sms_routing/{sms_routing_id} Update an SMS routing entry
SMS Routing DELETE /sms_routing/{sms_routing_id} Delete an SMS routing entry
SMS Routing GET /sms_routing/{sms_routing_id}/data Get list of SMS routing data
SMS Routing POST /sms_routing/{sms_routing_id}/data Create SMS routing data
SMS Routing PATCH /sms_routing/{sms_routing_id}/data/{sms_routing_data_id} Update SMS routing data
SMS Routing DELETE /sms_routing/{sms_routing_id}/data/{sms_routing_data_id} Delete SMS routing data

PDP Context Definitions Additions

The following endpoints for configuration of PDP Context Definitions have been improved to explain the request parameters better:

Topic Method Entrypoint Description
PDP Context Definitions POST /pdp_context_definition Create PDP Context Definition
PDP Context Definitions PATCH /pdp_context_definition/{definition_id} Update a PDP Context Definition

Updates

The following entrypoints for endpoint and organisation prepaid balance have been updated:

Topic Method Entrypoint Description
Endpoint GET /endpoint/{endpoint_id}/balance Updated Endpoint balance description
Prepaid Balance GET /organisation/{org_id}/balance Updated prepaid balance of given organisation

Additions

The following endpoints for retrieval of statistics have been added:

Topic Method Entrypoint Description
Statistics GET /endpoint/{endpoint_id}/stats/daily Endpoint Usage and Costs Statistics per day
Statistics GET /stats/daily Organisation Usage and Costs Statistics per day for the current month
Statistics GET /organisation/{org_id}/stats/daily Organisation Usage and Costs Statistics per day
Statistics GET /sim/{id}/stats SIM Usage and Costs Statistics
Statistics GET /sim/{id}/stats/daily SIM Usage and Costs Statistics per day

In 1.2.9, the following MNO-only operations have been added:

Network Coverage Additions

Topic Method Entrypoint Description
Network Coverage GET /network_coverage Retrieve Network Coverage Collection
Network Coverage GET /network_coverage/{coverage_id}/coverage Retrieve Network Coverage Data
Network Coverage PUT /network_coverage/{coverage_id}/coverage/{operator_id} Add Operator to Network Coverage
Network Coverage PATCH /network_coverage/{coverage_id}/coverage/{operator_id} Update VPLMN Status in Network Coverage Configuration

Operator Additions

Topic Method Entrypoint Description
Operator GET /operator List Operators
Operator POST /operator Add an Operator
Operator PATCH /operator/{operator_id} Update an Operator
Operator DELETE /operator/{operator_id} Delete an Operator
Operator GET /operator/{operator_id}/{operator_data} Retrieve Related Data of an Operator
Operator POST /operator/{operator_id}/{operator_data} Update Related Data of an Operator
Operator DELETE /operator/{operator_id}/{operator_data}/{op_data_id} Delete Related Data of an Operator

PDP Context Definitions

Topic Method Entrypoint Description
PDP Context Definitions GET /pdp_context_definition Retrieve Collection of PDP context definitions
PDP Context Definitions POST /pdp_context_definition Create PDP Context Definition
PDP Context Definitions PATCH /pdp_context_definition/{definition_id} Update a PDP Context definition
PDP Context Definitions DELETE /pdp_context_definition/{definition_id} Delete a PDP Context definition
Tariffs PUT /tariff/{tariff_id}/pdp_context_definition/{definition_id} Assign PDP Context Definition to Tariff
Tariffs DELETE /tariff/{tariff_id}/pdp_context_definition/{definition_id} Unassign PDP Context Definition from Tariff

Inter Operator Tariff

Topic Method Entrypoint Description
Inter Operator Tariff GET /iot List Inter Operator Tariffs
Inter Operator Tariff POST /iot Add new Inter Operator Tariff rate
Inter Operator Tariff PATCH /iot/{id} Get Inter Operator Tariff rate by ID

Tariff Profile Custom Rates

Topic Method Entrypoint Description
Tariff Profile GET /tariff_profile/{id} List Tariff Profile Details by ID

Tariff Profiles now contain applied custom rates for included tariffs where they exist.

Version 1.2.7 Additions

The following API requests have been added

Topic Method Entrypoint Description
Billing GET /organisation/{org_id}/billing Retrieve a list of available billing periods (by default the calendar month) for which billing data is available. This will also include the current billing period.
Billing GET /organisation/{org_id}/billing/{billing_id} Retrieve a list of available billing periods for an organisation.
Quota Management GET /endpoint/{endpoint_id}/quota/data Retrieve the current data quota status for an endpoint
Quota Management POST /endpoint/{endpoint_id}/quota/data A new data quota can be set for an endpoint
Quota Management GET /endpoint/{endpoint_id}/quota/sms Retrieve the current SMS quota status for an endpoint
Quota Management POST /endpoint/{endpoint_id}/quota/sms A new SMS quota can be set for an endpoint
Prepaid Balance Management GET /endpoint/{endpoint_id}/balance Retrieve the current balance for an endpoint
Prepaid Balance Management POST /endpoint/{endpoint_id}/balance The balance of an endpoint can be updated, either by adding or subtracting a certain amount
Prepaid Balance Management DELETE /endpoint/{endpoint_id}/balance The prepaid balance can be reset, meaning it will start with a value of zero

Quota Management Events

If a threshold percentage has been submitted and a data quota or SMS quota was created, and the remaining volume goes below that threshold. Threshold Reached events are created for data and SMS thresholds.

In case the data or SMS quota volume is completely depleted, Quota Used Up events are created.

Endpoint Events can be retrieved at /endpoint/{endpoint_id}/event.

Prepaid Balance Management

Balance management is activated in the service profile and will be applied to all endpoints assigned to that service profile. Initial balance is zero and endpoints are therefore blocked from any tele-services.