Detecting Azure Orchestration Errors with CloudForms

The Request Queue

The CloudForms service request queue shows automation requests like VM and Service Provision, however failure feedback for the latter is pretty basic. Further digging and log scraping to determine the root cause is usually necessary.

This is certainly the case with Cloud Orchestration Templates, including Microsoft Azure, this being the standard failure message:

before

The new azure-armrest Gem used to integrate ManageIQ/CloudForms with Azure, includes code to grab Azure events for given orchestration stacks. Using this Gem, we can get any errors and pass them back to the request.

The Ruby

To achieve the desired result, changes to two methods are required.

Check Provisioned

Extending the check_provisioned State method, we connect to Azure and get any error events here:
/ Cloud / Orchestration / Provisioning / StateMachines / Methods / check_provisioned

Connect to Azure's REST API:

conf = Azure::Armrest::ArmrestService.configure(
    :client_id  => service.orchestration_manager.authentication_userid,
    :client_key => service.orchestration_manager.authentication_password,
    :tenant_id  => service.orchestration_manager.uid_ems
)

Get the Stack Name, Resource Group and the Correlation Id:

stack_name      = service.stack_name
resource_group  = service.options[:create_options][:resource_group]
raw_stack       = Azure::Armrest::TemplateDeploymentService.new(conf).get(stack_name, resource_group)
correlation_id  = raw_stack.properties.correlation_id

Then use the Event Service to list all the events based on the filter:

event_service   = Azure::Armrest::Insights::EventService.new(conf)
date            = (Time.now - 86400).httpdate
filter          = "eventTimestamp ge #{date} and correlationId eq #{correlation_id}"
select          = 'correlationId, Properties'
events          = event_service.list(filter, select)

Finally, write the result to a state variable and to the request user message:

@request_message = @request_message.truncate(255) if @request_message.respond_to?(:truncate)
$evm.set_state_var('request_message', @request_message)
task.miq_request.user_message = request_message

The full method is here.

Update Service Provision Status

Extending the update_serviceprovision_status State On Entry/Exit/Error method is also required, so that the request Azure error message is not overwritten, here:
/ Custom / Cloud / Orchestration / Provisioning / StateMachines / Provision / update_serviceprovision_status

Update the service_template_provision_task message:

request_message = $evm.get_state_var('request_message')
prov.message = request_message

The full method is here.

The Result

Now, when an Azure stack error is detected, the error event should be shown in the Last Message field of the service request. Here are some examples:

after