I have a service which leverages Azure’s Service Bus Queues. Clients post messages with session into a request queue and then wait/block on the response coming back via a response queue. I have a number of Azure Persistent VMs which each have a windows service which monitors the request queue and once they have work it takes about 3 seconds for them to process the request and queue a response into to the response queue for the waiting client.
My clients are server processes themselves which run on servers in my Dallas datacenter, the VMs are part of the Azure Preview and must run within the West US datacenter which I suspect is somewhere in Washington state. Now the question is where do I run my Azure Service Bus Queues? Since the South Central US datacenter is in San Antonio, Texas I am fortunate to have something fairly close, or do I put the Service Bus Namespace in the same datacenter as my VMs, West US?
I could really see this going either way so I just decided to test by creating a Service Bus Namespace and queues in each datacenter (South Central US and West US) and modified my configurations on the client and server processes to use each queue as part of my test.
The image below documents my test results. It appears that by moving the Service Bus Namespace into the South Central US data center, closer to the clients, I can minimize the entire round trip latency by almost 30-40%. These numbers are not the true network latency because included in these numbers are the time it takes to queue and dequeue the messages however there was only ever one request in flight and each party was blocking waiting on the message in the queue. So all things being equal we can assume this test is valid. So then why the big difference? I suspect it is due to the fact for most of the journey the message traveled within Azure’s backbone and even with the detour down south a few hundred miles before heading back up north to Washington (and again on the way back) the latency must be much lower than what I can achieve going directly to Washington state. Of course the other option is there is an issue within the West US datacenter between VMs (IAAS) and Service Bus (PAAS) servers but I think that is highly unlikely but will keep monitoring.