Streamlining dynamic video generation

SoundCloud empowers artists and fans to connect and share through music. Founded in 2007, SoundCloud is an artist-first platform empowering artists to build and grow their careers by providing them with the most progressive tools, services, and resources. With over 375 million tracks from 40 million artists, the future of music is SoundCloud.

The Challenge: Streamlining Dynamic Video Generation

In the recent months, SoundCloud engineers have been working on workflows around building dynamic videos based on user stats. Building video generation and transcoding workflows is challenging. Workflows are split into many steps that generate, download, upload, and combine files together. As these steps take time, it's ideal to perform as much work in parallel as possible.

This requires significant manual effort in writing workers, handling retries, and managing the complexity of state across multiple steps, to name just a few. Traditional queuing services or tools like SQS or Bull require all of this additional work.

The goal was to find a solution that simplifies these challenges and allows engineers to focus on the business problem rather than infrastructure concerns.

Finding a better queue: a reliability layer

SoundCloud's decision to adopt Inngest was driven by their familiarity with queueing services and a desire to find a better way to handle workflows. Inngest stood out as a solution due to its use of HTTP as a transport for job creation, simplifying the abstraction of heavy lifting associated with queuing services.

Before settling on Inngest, the SoundCloud team explored alternatives such as the new version of Bull and other abstraction layers built on top of SQS. The evaluation included considerations for child queues/workers and the desire to find solutions that minimize the need for engineers to manage complex infrastructure. The focus was on efficiency and avoiding unnecessary manual effort in cloud-native application development.

In the end, Inngest's ability to handle queuing, orchestration, and retry logic enabled SoundCloud developers to focus on code with less overhead and more efficiency. They were searching for a better queue and found a reliability layer for their workflows.

Benefits

The adoption of Inngest resulted in several positive outcomes for SoundCloud:

1. Increased Productivity: With Inngest, complex workflows can be defined completely in code, abstracting away all infrastructure. This enables individual developers to be more efficient and productive without relying on other infrastructure engineers or spending time on the problem of infrastructure-as-code. This resulted in increased productivity as developers could read the code and trust that Inngest handled the underlying complexities of asynchronous calls, queuing, logging, and retries.
2. Reduced Context Switching: Before Inngest, when building systems like this, a developer would have to retain a large mental model of an entire system. This includes infrastructure (queues, scheduling, IaC), workers (polling, concurrency, orchestration) and application state; all in addition to the business logic itself.
   
   With Inngest, the mental model became more focused on the function code itself, reducing the mental picture and context switching that each developer needed to be concerned with. This helped streamline the development process.
3. Bus Factor Considerations: The use of complex systems can lead to a high bus factor, making it challenging for new engineers to understand and maintain the code. Inngest's simplification contributed to a more linear and readable codebase, addressing concerns related to knowledge transfer and team collaboration.

Conclusion

SoundCloud's adoption of Inngest for new video workflows brought about efficiency, reduced complexity, and increased productivity of the dev team. The case study emphasizes the importance of choosing tools that abstract away the overhead of infrastructure concerns, allowing engineers to focus on solving business problems. Inngest's features, including automatic handling of retries and simplification of complex workflows, gave SoundCloud's team the confidence to build their features more quickly with a small team.

If you're interested in learning how Inngest can help your team, reach out to us to chat with an expert.