Recently our mission-critical systems that rely on Azure Functions would occasionally stop responding. While we were glad to learn we weren’t alone, there still is no solution.

Our functions are triggered by Azure Storage Account Queues. So each time a message is added to the queue, it triggers the function. Below, a new message in some-queue triggers SomeJob:

[FunctionName("SomeJob")]
public static void SomeJob([QueueTrigger("some-queue")] string json, TraceWriter log)
{
    // Parse JSON
    var command = JsonConvert.DeserializeObject<SomeCommand>(json);

    // Execute command
    ...
}

As can be seen above, these are concise entry points into our application that allow us to rapidly make changes and deploy via Azure DevOps. While it didn’t happen often, these needed to be reliable. We found that restarting the App Service fixed the queue trigger. We had a monitor service that kept track of the queue size, so Application Insights notified us if these ever went down. This allowed for quick responses, but we didn’t want to worry about babysitting these.

Our temporary solution while we figured out what to do about Azure Functions: was to write another Azure Function. Luckily we had no issues with Timer triggered Azure functions (ones that run on a schedule). Along with the Azure Resource Management API, I wrote a function that:

1. Checks the current message count of the queue
2. Restart if queue exceeds N, a threshold that indicated messages were building up
3. If queue is normal, do nothing

[FunctionName("Health")]
public static void SomeJob([TimerTrigger("*/5 * * * *")] TimerInfo timer, TraceWriter log)
{
    // Get message count using Microsoft.Azure.Management.ResourceManager
    var messageCount = GetQueue().MessageCount;

    if (messageCount > threshold)
    {
        // Azure App Service from Microsoft.Azure.Management.ResourceManager
        appService.Restart()
    }
}