Track all function failures in Datadog
Your functions may fail from time to time. Inngest provides a way to handle all failed functions in a single place. This can enable you to send metrics, alerts, or events to external systems like Datadog or Sentry for all of your Inngest functions.
This page provides an example of tracking all function failures using Datadog's Events API to send all failures the Datadog event stream. You could replace Datadog with whatever system you use for monitoring and alerting.
Quick Snippet
Here is a basic function that uses the internal "inngest/function.failed" event. This event is triggered whenever any single function fails in your Inngest environment.
import { client, v1 } from "@datadog/datadog-api-client";
import { inngest } from "./client";
const configuration = client.createConfiguration();
const apiInstance = new v1.EventsApi(configuration);
export default inngest.createFunction(
{
name: "Send failures to Datadog",
id: "send-failed-function-events-to-datadog"
},
{ event: "inngest/function.failed" },
async ({ event, step }) => {
// This is a normal Inngest function, so we can use steps as we normally do:
await step.run("send-event-to-datadog", async () => {
const error = event.data.error;
// Create the Datadog event body using information about the failed function:
const params: v1.EventsApiCreateEventRequest = {
body: {
title: "Inngest Function Failed",
alert_type: "error",
text: `The ${event.data.function_id} function failed with the error: ${error.message}`,
tags: [
// Add a tag with the Inngest function id:
`inngest_function_id:${event.data.function_id}`,
],
},
};
// Send to Datadog:
const data = await apiInstance.createEvent(params);
// Return the data to Inngest for viewing in function logs:
return { message: "Event sent successfully", data };
});
}
);
An example failure event payload:
{
"name": "inngest/function.failed",
"data": {
"error": {
"__serialized": true,
"error": "invalid status code: 500",
"message": "taylor@ok.com is already a list member. Use PUT to insert or update list members.",
"name": "Error",
"stack": "Error: taylor@ok.com is already a list member. Use PUT to insert or update list members.\n at /var/task/.next/server/pages/api/inngest.js:2430:23\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async InngestFunction.runFn (/var/task/node_modules/.pnpm/inngest@2.6.0_typescript@5.1.6/node_modules/inngest/components/InngestFunction.js:378:32)\n at async InngestCommHandler.runStep (/var/task/node_modules/.pnpm/inngest@2.6.0_typescript@5.1.6/node_modules/inngest/components/InngestCommHandler.js:459:25)\n at async InngestCommHandler.handleAction (/var/task/node_modules/.pnpm/inngest@2.6.0_typescript@5.1.6/node_modules/inngest/components/InngestCommHandler.js:359:33)\n at async ServerTiming.wrap (/var/task/node_modules/.pnpm/inngest@2.6.0_typescript@5.1.6/node_modules/inngest/helpers/ServerTiming.js:69:21)\n at async ServerTiming.wrap (/var/task/node_modules/.pnpm/inngest@2.6.0_typescript@5.1.6/node_modules/inngest/helpers/ServerTiming.js:69:21)"
},
"event": {
"data": { "billingPlan": "pro" },
"id": "01H0TPSHZTVFF6SFVTR6E25MTC",
"name": "user.signup",
"ts": 1684523501562,
"user": { "external_id": "6463da8211cdbbcb191dd7da" }
},
"function_id": "my-gcp-cloud-functions-app-hello-inngest",
"run_id": "01H0TPSJ576QY54R6JJ8MEX6JH"
},
"id": "01H0TPW7KB4KCR739TG2J3FTHT",
"ts": 1684523589227
}
More context
Check the resources below to learn more about building email sequences with Inngest.