• Home
/
• Exploring m3ter - Getting to Know the Platform

Getting Usage Data into m3ter

Designed as a data-first solution from the outset, the m3ter platform helps you ingest usage data flexibly and easily, and with the minimum of ongoing effort. This topic provides a quick reference starter explaining some of the options available for getting your usage data into the platform.

Usage Data

We’ve built our platform to handle high volumes of usage events, and we encourage customers to send rich, granular usage data because this allows us to provide you with more flexibility and value from the data after ingestion.

Measurements are the individual units of usage data in m3ter, and their schema and data types are defined by Meters. You can find out more about Meters and the structure of Measurements in the Reviewing Meter Options topic. However, to demonstrate using a simple example - suppose you run a SaaS platform that provides messaging services via API, then you might define a Meter that collects usage data for the number of requests, the region in which they occurred, and the category of action performed:

1
{
2
  "uid": "705b30d2-4a79-43f3-9aae-a6ebe3894a2b",
3
  "meter": "api_requests",
4
  "account": "your_customer_ref_123456789",
5
  "ts": "2023-01-02T03:04:05.000Z",
6
  "measure": {
7
    "requests": 21
8
  },
9
  “what”: {
10
    “action”: “sendMessage”
11
  },
12
  “where”: {
13
    “region”: “eu-west-1”
14
  }
15
}
16

There are two endpoints for ingesting usage data - Ingest REST API and File Upload. We also offer multiple client-side options to help you deliver your usage data reliably, quickly, and cost-effectively

Ingest API

Our Ingest REST API is the primary mechanism for ingesting your usage data into the platform. It’s easy to use, and is designed with scale in mind. See the API Calls for Data Ingest topic and the Submit Measurements section in our API Reference documentation.

Some request rate and payload size limits are imposed on data submissions to the Ingest API. For details on these limits and guidance on optimizing your data submissions, see Ingest Limits and Optimizing Submissions.

File Upload

If your existing systems and data flow are better suited to uploading a file containing usage data, perhaps at the end of a batch operation, we offer a File Upload service. The process is simple - make a call to the API to generate a temporary upload URL, then upload your file and we’ll do the rest. Any errors are reported to you via the normal Events and Notifications service. See our File Upload Service API topic for more information.

Usage Data Ingest Process

The usage data ingest process has two stages:

• Pre-enrichment. This stage is kept as simple as possible to avert the possibility of data loss:

  • It is synchronous and is the level at which any API response to a usage data measurement submission refers.

  • Data validation checks are not performed at this stage and a usage data submission might not be immediately rejected.

• Enrichment. This stage is where data validation checks are made:

  • It is asynchronous.

  • A usage data submission that was accepted for the pre-enrichment stage might be rejected at this stage because it fails a validation check that is performed during data enrichment.

m3ter Ingest Agent

We’ve made every effort to make ingesting your data via our API as easy as possible, but integrating with any API requires the development and maintenance of software to handle authentication, batching, retries, and so on. For that reason, we developed the m3ter Ingest Agent, which:

• Handles authentication, including automatic refreshing of the session bearer token.

• Automatically batches measurements to reduce the number of calls made to our API.

• Queues measurements locally if there are any issues, and retries automatically.

• Can take input in any format and via many protocols, translating the input to m3ter measurement format.

• Can manipulate data before sending to the m3ter API, which can be useful if you have usage in multiple formats and want to unify them prior to ingest.

• Can provide a full audit of what has been sent to the m3ter API.

• Can be run continuously (standalone or as a sidecar to your applications), on a schedule, or as an action from a trigger in your workflow.

The Ingest Agent is available as a binary and a Docker container image.

Third-Party Integrations and Managed Ingest

Many customers have usage data already being passed between systems via streams and queues. We can often help these customers get up and running quickly without additional steps and infrastructure by ‘hooking into’ existing data streams, transforming them into m3ter measurement format and sending them to our API. This ingest capability is currently available for Amazon SNS, Amazon SQS, and Apache Kafka, with support for Customer Data Platforms (CDPs) such as Segment, and Reverse ETL platforms planned.

Additionally, we offer Managed Ingest solutions that typically involve m3ter pulling usage data from a set endpoint or location (like an S3 bucket) on the customer’s behalf.

SDKs (Coming soon!)

m3ter SDKs will reduce the development effort associated with integrating your applications with m3ter, and help ensure efficient behavior with the API. They’ll be available in Java and .NET, and will maintain feature parity with our public APIs.

Error Handling

m3ter gives you complete visibility of exceptions that occur during ingest. There are two feedback processes, depending on the exception:

• Those that can be detected at the point of submission - for example, authorization failures, malformed payloads, and missing mandatory fields like the measurement timestamp. These are indicated clearly and immediately in the API response.

• Errors that occur in the ingest and enrichment stage - for example, validation against your Meter configurations and failures in Derived Field calculations. These are reported asynchronously via the Events service. Ingest Error events contain the error detail, the original measurement submitted, and additional information to assist with troubleshooting. You can then set up Notifications to be triggered when these error Events occur for manual or automated follow-up.

See the Reviewing and Resolving Ingest Events topic for further information.

Idempotency and Deduplication

No system is immune from failures, especially when transferring data across the Internet, and there are many reasons measurements might get lost or duplicated on their way into the m3ter platform. To handle this, a key feature of the m3ter ingest service is that sending a measurement is an idempotent operation within a 35 day window. This means that if a measurement with the same UID arrives multiple times in that time-frame, we'll only ingest it once.

If you’re not sure whether a network connectivity issue or application crash affected measurements being sent, you can simply resend them with their original UIDs and we’ll de-duplicate and only store the missing ones.

General Recommendations

There are some general recommendations worth noting for usage data ingest:

• More not Less. Send us more data rather than less. Also, where possible enrich the usage data you send us, for example by sending data more frequently, or by adding more data dimensions and metadata. The richer the data you send us, the more powerful you’ll find m3ter since we’ll be able to provide better analytics on your ingested data.

• Queue Usage Events. If your usage events are short-lived or not retained after being submitted, we recommend putting them on a queue, such as Amazon SQS, Azure Storage Queues, or RabbitMQ, within your environment first. Then, read from that queue when submitting to the m3ter Ingest API. This adds a layer of protection against data loss due to network or software issues.

FAQs

Can you fetch my usage data instead of me submitting it?

Yes. Our Ingest Agent can poll external systems and submit usage data to the m3ter API. Various sources and formats are supported out-of-the-box, but we can also help develop custom collectors should one be required.

Is it possible to delete or modify usage data?

Not directly. Measurements are immutable once ingested, but it’s possible to make adjustments to correct aggregated data by submitting corrective measurements (either positive or negative). See the Submitting Test Usage Data and Dates in m3ter topics for more information.

Can I export my data?

Yes. You can view, filter, aggregate, and export usage data via our Data Explorer, either by using API calls or through the UI.

In the future, we'll be introducing additional features to allow you to export larger volumes of data. For now if you require bulk export, you can submit a request to m3ter Support to assist you

What happens if I need to add fields to my Meter or remove fields after I've already sent data?

If you send measurements containing data fields that are not yet configured in your Meter, we still store the data. When you create a Data Field with the same code field, it will become linked and available for rating. Removing a Data Field from a Meter does not delete the existing measurements, but it becomes unavailable for other m3ter features.

Can I send you any Personal Data in usage?

No. Don’t put any Personal Data (sometimes referred to as PI or PII) in the usage data. Instead, use anonymous unique identifiers that map to Personal Data in your CRM or other systems. A limited amount of Personal Data can be used in the Account object. Please see section 4.2 “Customer responsibility for Customer Data and Authorised User Accounts” in our Terms of Service and section 4 “Data Retention” of m3ter’s Data Protection Agreement for more details.

Data Ingest Overview - Diagram

This diagram gives an overview of how usage data fits into the wider m3ter platform:

Next: Integrations for m3ter