Mitigating System Congestion in Telestream Vantage
Many Vantage users eventually encounter system congestion—a situation where too many jobs hit the system at once, overwhelming available compute resources or session limits. This congestion can arise from several common scenarios:
Watched Folders: A user drag-and-drops a large batch of files.
High-volume API Submissions: A client bulk-orders jobs via an external system.
Job Restarts: A user restarts a large number of failed or intentionally stopped jobs through Vantage Workflow Designer or Web Job Status View.
Workorders with Many Jobs: One or more large, multi-job workorders get submitted.
The result is the same: too many jobs flood one or more workflows at once, pushing the system past its limits.
Why Congestion Matters
The consequences of congestion are almost always negative:
System throughput slows dramatically.
Backlogs build up, delaying processing.
Resource pre-allocation for certain services can create unexpected system issues.
Priority jobs can stall due to race conditions, preventing any jobs from completing.
Other users’ jobs get 'stuck', appearing unresponsive and prompting repeated resubmissions—which only worsen the congestion.
In business terms, this translates to:
User frustration and dissatisfaction.
Missed SLAs.
A perception that Vantage “can’t support the business.”
Practical Mitigations
The good news is that congestion can be mitigated—and often eliminated—through smart practices and workflow design. Some proven strategies include:
1. User Education
Encouraging operators to submit smaller batches of jobs, distributed over time, which leads to much more efficient resource utilization and smooths out peaks in volume. This applies to workflows which use either Workorders or Watched Folders.
In the real-world we frequently see cases where users prep multiple jobs or workorders then submit them at the end of the day, and walk out the door. As a consequence they don't see the impact of this practice, which other users and system admins have to deal with.
Also, help users understand that jobs are rarely 'stuck' if they show in active state; rather they are awaiting resources to complete and given some patience they will generally complete. They should be discouraged from resubmitting 'stuck' jobs as this generally just adds mode load to the system and exacerbates the problem.
2. Flow Control via SDK Service
When external systems submit jobs through the SDK Service, a rate limiting or flow control mechanism should be in place. We’ve implemented very effective closed-loop control solutions that dynamically manage job submission rates based on current system load and available resources.
It is also important to be cognizant of what other workflows may be active in the Vantage domain and ensure that some resources remain available to service jobs in them.
3. Rate Limiting Inside Workflows
When large batches are unavoidable, workflows themselves can be designed to throttle throughput. We’ve developed multiple workflow-level strategies to smooth out submission spikes and mitigate potential resource saturation.
4. Distributing Jobs Over Time
Even if a large workorder is submitted and the jobs all start at once, workflows can be built to spread job execution over time.
One real-world example: a client needed to prepare multiple seasons of a show in a single workorder. By designing a workflow that staggered job execution, they could safely submit the full workorder on a Friday evening. Vantage then processed the jobs across the weekend in an orderly way—always keeping resources available for other tasks.
Regain Control of Your System
Do these scenarios sound familiar? Are congestion and backlog keeping your Vantage system from running at full potential?
We specialize in helping Vantage users design smarter workflows, implement flow control, and eliminate congestion bottlenecks.
👉 Let’s talk—so your workflows work for you, not against you.