2018-11-29 20:58:07,331 INFO nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Assigning node request 2018-11-29 20:58:07,331 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4]: Declining node request 300-0000331925 because node type(s) [ansible-fedora-28-1vcpu] not available 2018-11-29 20:58:07,334 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 20:58:07,341 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 20:58:07,341 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 20:58:07,341 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 20:58:07,342 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 20:58:07,343 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 20:58:07,343 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 20:58:07,343 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 20:58:07,346 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 20:58:07,348 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 20:58:07,348 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 20:58:07,349 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 20:58:07,351 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 20:58:07,362 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Removing handler for request 300-0000331925 2018-11-29 20:58:07,362 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 20:58:07,737 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000242494 (state: deleting, allocated_to: None) 2018-11-29 20:58:07,737 INFO nodepool.DeletedNodeWorker: Deleting deleting instance d0021aac-4e31-42c5-a6a6-b71097532a56 from rdo-cloud-tripleo 2018-11-29 20:58:07,742 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server d0021aac-4e31-42c5-a6a6-b71097532a56 2018-11-29 20:58:08,157 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.5276925563812256s 2018-11-29 20:58:08,175 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 20:58:08,253 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.0782313346862793s 2018-11-29 20:58:08,614 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:09,001 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.38652467727661133s 2018-11-29 20:58:09,005 ERROR nodepool.NodeLauncher-0000318658: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318658: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 157, in _launchNode auto_ip=self.pool.auto_floating_ip) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 356, in waitForServer reuse=False, timeout=timeout) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6881, in wait_for_server nat_destination=nat_destination) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6898, in get_active_server "Error in creating the server", extra_data=dict(server=server)) openstack.exceptions.SDKException: Error in creating the server 2018-11-29 20:58:09,010 INFO nodepool.NodeLauncher-0000318658: Request 300-0000331819: Node 7937734d-acc0-459e-a97b-897b2f0c2482 scheduled for cleanup 2018-11-29 20:58:09,016 INFO nodepool.NodeLauncher-0000318658: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318658 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318658 2018-11-29 20:58:09,016 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 20:58:10,328 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:10,597 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.5806734561920166s 2018-11-29 20:58:10,598 ERROR nodepool.NodeLauncher-0000318658: Request 300-0000331819: Launch attempt 2/3 failed for node 0000318658: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 20:58:10,598 INFO nodepool.NodeLauncher-0000318658: Quota exceeded, invalidating quota cache 2018-11-29 20:58:10,599 INFO nodepool.NodeLauncher-0000318658: Aborting node 0000318658 due to quota failure 2018-11-29 20:58:10,741 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.4120967388153076s 2018-11-29 20:58:11,641 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:12,085 INFO nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Assigning node request 2018-11-29 20:58:12,089 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 20:58:12,282 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.19297218322753906s 2018-11-29 20:58:12,282 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 20:58:12,283 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 20:58:12,458 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.17516255378723145s 2018-11-29 20:58:12,461 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 20:58:12,462 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Accepting node request 300-0000331925 2018-11-29 20:58:12,520 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 20:58:12,565 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 20:58:12,616 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 20:58:12,616 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 20:58:12,626 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318660 for request 300-0000331925 2018-11-29 20:58:12,633 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 20:58:12,633 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 20:58:12,635 INFO nodepool.NodeLauncher-0000318660: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318660 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318660 2018-11-29 20:58:12,636 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 20:58:12,794 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318659 (state: deleting, allocated_to: None) 2018-11-29 20:58:12,794 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 7937734d-acc0-459e-a97b-897b2f0c2482 from vexxhost-ansible-network-mtl1 2018-11-29 20:58:12,803 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 7937734d-acc0-459e-a97b-897b2f0c2482 2018-11-29 20:58:12,806 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1646015644073486s 2018-11-29 20:58:12,826 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318658 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 20:58:12,826 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 20:58:12,830 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318658, state=deleting, external_id=None 2018-11-29 20:58:13,528 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.8925485610961914s 2018-11-29 20:58:13,529 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeDeleteServers (queue 0) 2018-11-29 20:58:13,529 ERROR nodepool.NodeLauncher-0000318660: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318660: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 20:58:13,533 INFO nodepool.NodeLauncher-0000318660: Quota exceeded, invalidating quota cache 2018-11-29 20:58:13,533 INFO nodepool.NodeLauncher-0000318660: Aborting node 0000318660 due to quota failure 2018-11-29 20:58:14,190 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 20:58:14,608 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeDeleteServers in 1.0791404247283936s 2018-11-29 20:58:14,609 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:15,115 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5060896873474121s 2018-11-29 20:58:15,117 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318659, state=deleting, external_id=7937734d-acc0-459e-a97b-897b2f0c2482 2018-11-29 20:58:15,383 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1912050247192383s 2018-11-29 20:58:16,756 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:17,936 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.1800060272216797s 2018-11-29 20:58:17,954 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:18,001 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 20:58:18,002 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 20:58:18,005 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 20:58:18,006 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 20:58:18,006 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 20:58:18,007 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 20:58:18,008 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 20:58:18,008 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 20:58:18,008 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 20:58:18,009 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 20:58:18,009 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 20:58:18,009 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 20:58:18,010 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 20:58:18,010 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 20:58:18,936 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.981842041015625s 2018-11-29 20:58:21,414 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 20:58:22,638 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 20:58:22,695 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2806475162506104s 2018-11-29 20:58:22,859 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2208249568939209s 2018-11-29 20:58:22,860 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 20:58:22,860 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 20:58:22,978 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:23,056 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.19609689712524414s 2018-11-29 20:58:23,060 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:23,750 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6899325847625732s 2018-11-29 20:58:23,768 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 20:58:23,768 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 20:58:23,875 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 20:58:23,924 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 20:58:23,924 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 20:58:23,925 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 20:58:23,984 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318661 for request 300-0000331819 2018-11-29 20:58:23,997 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 20:58:23,997 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 20:58:23,998 INFO nodepool.NodeLauncher-0000318661: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318661 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318661 2018-11-29 20:58:23,999 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 20:58:24,100 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:25,220 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 2.241903066635132s 2018-11-29 20:58:25,683 ERROR nodepool.NodeDeleter: Exception deleting instance 33034fa9-59e6-48d7-88ee-cd4f0bb01e35 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 33034fa9-59e6-48d7-88ee-cd4f0bb01e35 deletion 2018-11-29 20:58:25,708 ERROR nodepool.NodeDeleter: Exception deleting instance 75e44b32-409c-43fd-9ec9-fa590b96b917 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 75e44b32-409c-43fd-9ec9-fa590b96b917 deletion 2018-11-29 20:58:25,840 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.7394795417785645s 2018-11-29 20:58:25,963 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.9640016555786133s 2018-11-29 20:58:25,963 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServers (queue 0) 2018-11-29 20:58:26,257 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServers in 0.2935211658477783s 2018-11-29 20:58:26,265 DEBUG nodepool.NodeLauncher-0000318661: Waiting for server 64cb0a17-bf48-496e-b6b1-663737e2d60b for node id: 0000318661 2018-11-29 20:58:28,700 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318660 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 20:58:28,700 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 20:58:28,708 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318660, state=deleting, external_id=None 2018-11-29 20:58:28,711 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 20:58:28,711 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 20:58:28,712 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 20:58:28,713 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 20:58:28,714 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 20:58:28,714 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 20:58:28,715 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 20:58:28,715 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 20:58:28,715 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 20:58:28,716 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 20:58:28,717 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 20:58:28,717 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 20:58:28,718 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 20:58:28,718 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 20:58:28,723 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 20:58:28,815 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211297 (state: deleting, allocated_to: None) 2018-11-29 20:58:28,815 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 75e44b32-409c-43fd-9ec9-fa590b96b917 from rdo-cloud-tripleo 2018-11-29 20:58:28,820 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 75e44b32-409c-43fd-9ec9-fa590b96b917 2018-11-29 20:58:28,833 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211288 (state: deleting, allocated_to: None) 2018-11-29 20:58:28,833 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 33034fa9-59e6-48d7-88ee-cd4f0bb01e35 from rdo-cloud-tripleo 2018-11-29 20:58:28,838 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 33034fa9-59e6-48d7-88ee-cd4f0bb01e35 2018-11-29 20:58:30,058 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3349859714508057s 2018-11-29 20:58:30,074 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 1) 2018-11-29 20:58:30,295 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.2192835807800293s 2018-11-29 20:58:30,298 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 20:58:30,361 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.06343626976013184s 2018-11-29 20:58:30,968 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:31,235 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:31,268 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:31,613 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.34531211853027344s 2018-11-29 20:58:31,618 ERROR nodepool.NodeLauncher-0000318661: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318661: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 157, in _launchNode auto_ip=self.pool.auto_floating_ip) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 356, in waitForServer reuse=False, timeout=timeout) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6881, in wait_for_server nat_destination=nat_destination) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6898, in get_active_server "Error in creating the server", extra_data=dict(server=server)) openstack.exceptions.SDKException: Error in creating the server 2018-11-29 20:58:31,621 INFO nodepool.NodeLauncher-0000318661: Request 300-0000331819: Node 64cb0a17-bf48-496e-b6b1-663737e2d60b scheduled for cleanup 2018-11-29 20:58:31,627 INFO nodepool.NodeLauncher-0000318661: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318661 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318661 2018-11-29 20:58:31,627 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 20:58:31,960 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9910428524017334s 2018-11-29 20:58:32,034 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7985560894012451s 2018-11-29 20:58:33,207 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.5797536373138428s 2018-11-29 20:58:33,208 ERROR nodepool.NodeLauncher-0000318661: Request 300-0000331819: Launch attempt 2/3 failed for node 0000318661: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 20:58:33,208 INFO nodepool.NodeLauncher-0000318661: Quota exceeded, invalidating quota cache 2018-11-29 20:58:33,208 INFO nodepool.NodeLauncher-0000318661: Aborting node 0000318661 due to quota failure 2018-11-29 20:58:33,997 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318662 (state: deleting, allocated_to: None) 2018-11-29 20:58:33,997 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 64cb0a17-bf48-496e-b6b1-663737e2d60b from vexxhost-ansible-network-mtl1 2018-11-29 20:58:34,003 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 64cb0a17-bf48-496e-b6b1-663737e2d60b 2018-11-29 20:58:34,003 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeDeleteServers (queue 0) 2018-11-29 20:58:34,686 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeDeleteServers in 0.6829423904418945s 2018-11-29 20:58:34,689 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 20:58:34,888 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.19854378700256348s 2018-11-29 20:58:34,888 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 20:58:34,889 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 20:58:35,023 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13381671905517578s 2018-11-29 20:58:35,026 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 20:58:35,026 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 20:58:35,080 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 20:58:35,127 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 20:58:35,175 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 20:58:35,175 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 20:58:35,176 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 20:58:35,185 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318663 for request 300-0000331925 2018-11-29 20:58:35,194 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 20:58:35,195 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 20:58:35,196 INFO nodepool.NodeLauncher-0000318663: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318663 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318663 2018-11-29 20:58:35,197 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 20:58:36,088 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 20:58:36,914 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.717235803604126s 2018-11-29 20:58:36,915 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 1) 2018-11-29 20:58:37,131 ERROR nodepool.NodeDeleter: Exception deleting instance 3f4f9477-ce7a-4bd1-af21-8ce7eedc74e8 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 3f4f9477-ce7a-4bd1-af21-8ce7eedc74e8 deletion 2018-11-29 20:58:37,320 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2323939800262451s 2018-11-29 20:58:37,336 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:37,616 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.7010633945465088s 2018-11-29 20:58:37,618 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServers (queue 0) 2018-11-29 20:58:37,618 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318662, state=deleting, external_id=64cb0a17-bf48-496e-b6b1-663737e2d60b 2018-11-29 20:58:37,973 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServers in 0.35428714752197266s 2018-11-29 20:58:37,979 DEBUG nodepool.NodeLauncher-0000318663: Waiting for server a3bcb985-e362-4826-be85-e5ec04a0368e for node id: 0000318663 2018-11-29 20:58:37,981 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:38,216 ERROR nodepool.NodeDeleter: Exception deleting instance 6a0c9d49-2789-4188-92f2-aa94b2491ce5 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 6a0c9d49-2789-4188-92f2-aa94b2491ce5 deletion 2018-11-29 20:58:38,413 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0746021270751953s 2018-11-29 20:58:38,912 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9303300380706787s 2018-11-29 20:58:39,274 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318661 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 20:58:39,274 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 20:58:39,282 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318661, state=deleting, external_id=None 2018-11-29 20:58:39,499 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 20:58:39,501 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 20:58:39,502 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 20:58:39,504 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 20:58:39,505 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 20:58:39,505 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 20:58:39,506 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 20:58:39,506 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 20:58:39,507 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 20:58:39,507 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 20:58:39,507 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 20:58:39,508 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 20:58:39,508 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 20:58:39,509 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 20:58:39,605 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211294 (state: deleting, allocated_to: None) 2018-11-29 20:58:39,605 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 6a0c9d49-2789-4188-92f2-aa94b2491ce5 from rdo-cloud-tripleo 2018-11-29 20:58:39,611 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 6a0c9d49-2789-4188-92f2-aa94b2491ce5 2018-11-29 20:58:39,611 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 20:58:39,655 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211289 (state: deleting, allocated_to: None) 2018-11-29 20:58:39,655 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 3f4f9477-ce7a-4bd1-af21-8ce7eedc74e8 from rdo-cloud-tripleo 2018-11-29 20:58:39,660 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 3f4f9477-ce7a-4bd1-af21-8ce7eedc74e8 2018-11-29 20:58:39,790 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318135 (state: used, allocated_to: 200-0000331919) 2018-11-29 20:58:39,790 INFO nodepool.DeletedNodeWorker: Deleting used instance None from ansible-statics 2018-11-29 20:58:39,794 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318135, state=deleting, external_id=None 2018-11-29 20:58:39,825 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.21358251571655273s 2018-11-29 20:58:39,828 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 20:58:40,045 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.21704578399658203s 2018-11-29 20:58:40,200 DEBUG nodepool.driver.static.StaticNodeProvider: Registered static node 35.199.20.133 2018-11-29 20:58:42,433 ERROR nodepool.NodeDeleter: Exception deleting instance b4853e6a-3115-451a-96cb-3c751068e393 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server b4853e6a-3115-451a-96cb-3c751068e393 deletion 2018-11-29 20:58:42,985 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:43,350 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 20:58:43,473 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.4873647689819336s 2018-11-29 20:58:43,481 ERROR nodepool.NodeLauncher-0000318663: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318663: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 157, in _launchNode auto_ip=self.pool.auto_floating_ip) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 356, in waitForServer reuse=False, timeout=timeout) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6881, in wait_for_server nat_destination=nat_destination) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6898, in get_active_server "Error in creating the server", extra_data=dict(server=server)) openstack.exceptions.SDKException: Error in creating the server 2018-11-29 20:58:43,485 INFO nodepool.NodeLauncher-0000318663: Request 300-0000331925: Node a3bcb985-e362-4826-be85-e5ec04a0368e scheduled for cleanup 2018-11-29 20:58:43,490 INFO nodepool.NodeLauncher-0000318663: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318663 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318663 2018-11-29 20:58:43,491 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 20:58:44,058 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:44,570 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2202305793762207s 2018-11-29 20:58:44,586 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:44,849 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318665 (state: deleting, allocated_to: None) 2018-11-29 20:58:44,849 INFO nodepool.DeletedNodeWorker: Deleting deleting instance a3bcb985-e362-4826-be85-e5ec04a0368e from vexxhost-ansible-network-mtl1 2018-11-29 20:58:44,858 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server a3bcb985-e362-4826-be85-e5ec04a0368e 2018-11-29 20:58:44,984 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211299 (state: deleting, allocated_to: None) 2018-11-29 20:58:44,984 INFO nodepool.DeletedNodeWorker: Deleting deleting instance b4853e6a-3115-451a-96cb-3c751068e393 from rdo-cloud-tripleo 2018-11-29 20:58:44,989 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server b4853e6a-3115-451a-96cb-3c751068e393 2018-11-29 20:58:45,201 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 20:58:45,214 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.1557600498199463s 2018-11-29 20:58:45,289 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 20:58:45,364 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 20:58:45,439 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 20:58:45,439 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 20:58:45,439 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 20:58:45,451 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318666 for request 300-0000331819 2018-11-29 20:58:45,455 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 20:58:45,458 INFO nodepool.NodeLauncher-0000318666: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318666 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318666 2018-11-29 20:58:45,520 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 2.028886556625366s 2018-11-29 20:58:45,521 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeDeleteServers (queue 1) 2018-11-29 20:58:45,520 ERROR nodepool.NodeLauncher-0000318663: Request 300-0000331925: Launch attempt 2/3 failed for node 0000318663: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 20:58:45,524 INFO nodepool.NodeLauncher-0000318663: Quota exceeded, invalidating quota cache 2018-11-29 20:58:45,524 INFO nodepool.NodeLauncher-0000318663: Aborting node 0000318663 due to quota failure 2018-11-29 20:58:45,643 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0568134784698486s 2018-11-29 20:58:45,656 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 20:58:45,726 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.06908726692199707s 2018-11-29 20:58:46,833 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeDeleteServers in 1.3120999336242676s 2018-11-29 20:58:46,836 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 20:58:47,280 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:47,739 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.4590315818786621s 2018-11-29 20:58:47,739 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-6c84cb2e-5c10-4089-be5d-63537876fe68) 2018-11-29 20:58:47,885 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 20:58:49,062 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 2.226428747177124s 2018-11-29 20:58:49,063 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 1) 2018-11-29 20:58:49,492 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.42861199378967285s 2018-11-29 20:58:49,494 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318665, state=deleting, external_id=a3bcb985-e362-4826-be85-e5ec04a0368e 2018-11-29 20:58:49,541 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServers (queue 0) 2018-11-29 20:58:49,900 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServers in 0.3595249652862549s 2018-11-29 20:58:49,915 DEBUG nodepool.NodeLauncher-0000318666: Waiting for server 8d473f7f-4b35-46a1-82ce-2d3f2274392e for node id: 0000318666 2018-11-29 20:58:50,386 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 20:58:50,389 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 20:58:50,391 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 20:58:50,391 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 20:58:50,392 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 20:58:50,395 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 20:58:50,395 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 20:58:50,396 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 20:58:50,396 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 20:58:50,396 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 20:58:50,397 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 20:58:50,398 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 20:58:50,398 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 20:58:50,398 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 20:58:50,601 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 20:58:50,931 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:51,797 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8655142784118652s 2018-11-29 20:58:51,986 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3845641613006592s 2018-11-29 20:58:52,001 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:53,014 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0099515914916992s 2018-11-29 20:58:54,922 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:55,566 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 20:58:55,566 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 20:58:55,597 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6753311157226562s 2018-11-29 20:58:55,602 ERROR nodepool.NodeLauncher-0000318666: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318666: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 157, in _launchNode auto_ip=self.pool.auto_floating_ip) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 356, in waitForServer reuse=False, timeout=timeout) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6881, in wait_for_server nat_destination=nat_destination) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6898, in get_active_server "Error in creating the server", extra_data=dict(server=server)) openstack.exceptions.SDKException: Error in creating the server 2018-11-29 20:58:55,605 INFO nodepool.NodeLauncher-0000318666: Request 300-0000331819: Node 8d473f7f-4b35-46a1-82ce-2d3f2274392e scheduled for cleanup 2018-11-29 20:58:55,610 INFO nodepool.NodeLauncher-0000318666: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318666 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318666 2018-11-29 20:58:55,611 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 20:58:55,673 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318663 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 20:58:55,673 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 20:58:55,677 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318663, state=deleting, external_id=None 2018-11-29 20:58:55,688 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318667 (state: deleting, allocated_to: None) 2018-11-29 20:58:55,688 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 8d473f7f-4b35-46a1-82ce-2d3f2274392e from vexxhost-ansible-network-mtl1 2018-11-29 20:58:55,694 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 8d473f7f-4b35-46a1-82ce-2d3f2274392e 2018-11-29 20:58:57,090 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4793708324432373s 2018-11-29 20:58:57,091 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeDeleteServers (queue 0) 2018-11-29 20:58:57,091 ERROR nodepool.NodeLauncher-0000318666: Request 300-0000331819: Launch attempt 2/3 failed for node 0000318666: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 20:58:57,094 INFO nodepool.NodeLauncher-0000318666: Quota exceeded, invalidating quota cache 2018-11-29 20:58:57,095 INFO nodepool.NodeLauncher-0000318666: Aborting node 0000318666 due to quota failure 2018-11-29 20:58:57,237 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:58:57,666 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.4286963939666748s 2018-11-29 20:58:58,015 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 20:58:58,100 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeDeleteServers in 1.009110450744629s 2018-11-29 20:58:59,394 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.379392385482788s 2018-11-29 20:58:59,410 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:00,597 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1840319633483887s 2018-11-29 20:59:01,234 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 20:59:01,235 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 20:59:01,236 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 20:59:01,236 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 20:59:01,238 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 20:59:01,239 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 20:59:01,239 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 20:59:01,240 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 20:59:01,240 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 20:59:01,243 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 20:59:01,243 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 20:59:01,243 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 20:59:01,244 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 20:59:01,244 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 20:59:02,108 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:02,774 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6656191349029541s 2018-11-29 20:59:02,776 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318667, state=deleting, external_id=8d473f7f-4b35-46a1-82ce-2d3f2274392e 2018-11-29 20:59:03,041 ERROR nodepool.NodeDeleter: Exception deleting instance 9c02645e-baeb-4e9e-8a50-62d4671f8c9c from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 9c02645e-baeb-4e9e-8a50-62d4671f8c9c deletion 2018-11-29 20:59:03,682 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:04,237 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.554964542388916s 2018-11-29 20:59:05,423 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 20:59:05,571 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 20:59:05,896 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.3244948387145996s 2018-11-29 20:59:05,896 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 20:59:05,897 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 20:59:06,025 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1285569667816162s 2018-11-29 20:59:06,028 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 20:59:06,029 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 20:59:06,143 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 20:59:06,199 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 20:59:06,199 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 20:59:06,199 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 20:59:06,209 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318668 for request 300-0000331925 2018-11-29 20:59:06,216 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 20:59:06,216 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 20:59:06,217 INFO nodepool.NodeLauncher-0000318668: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318668 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318668 2018-11-29 20:59:06,218 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 20:59:06,430 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318666 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 20:59:06,430 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 20:59:06,436 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318666, state=deleting, external_id=None 2018-11-29 20:59:06,651 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000242496 (state: deleting, allocated_to: None) 2018-11-29 20:59:06,651 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 9c02645e-baeb-4e9e-8a50-62d4671f8c9c from rdo-cloud-tripleo 2018-11-29 20:59:06,655 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 9c02645e-baeb-4e9e-8a50-62d4671f8c9c 2018-11-29 20:59:06,676 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318115 (state: used, allocated_to: 200-0000331922) 2018-11-29 20:59:06,676 INFO nodepool.DeletedNodeWorker: Deleting used instance None from ansible-statics 2018-11-29 20:59:06,679 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318115, state=deleting, external_id=None 2018-11-29 20:59:06,805 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.381415843963623s 2018-11-29 20:59:06,821 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 1) 2018-11-29 20:59:07,053 DEBUG nodepool.driver.static.StaticNodeProvider: Registered static node 35.221.28.96 2018-11-29 20:59:08,227 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.4030916690826416s 2018-11-29 20:59:08,241 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 20:59:08,456 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.21488428115844727s 2018-11-29 20:59:08,755 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 2.537459373474121s 2018-11-29 20:59:08,756 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServers (queue 0) 2018-11-29 20:59:09,179 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServers in 0.423020601272583s 2018-11-29 20:59:09,185 DEBUG nodepool.NodeLauncher-0000318668: Waiting for server 2a7089ab-753e-49c0-aaba-be29bdde925f for node id: 0000318668 2018-11-29 20:59:09,185 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:09,830 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:10,109 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.9232387542724609s 2018-11-29 20:59:10,376 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.5454850196838379s 2018-11-29 20:59:12,041 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 20:59:12,044 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 20:59:12,047 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 20:59:12,047 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 20:59:12,049 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 20:59:12,051 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 20:59:12,051 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 20:59:12,051 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 20:59:12,052 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 20:59:12,052 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 20:59:12,055 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 20:59:12,056 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 20:59:12,056 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 20:59:12,059 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 20:59:12,217 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318138 (state: used, allocated_to: 200-0000331921) 2018-11-29 20:59:12,217 INFO nodepool.DeletedNodeWorker: Deleting used instance None from ansible-statics 2018-11-29 20:59:12,221 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318138, state=deleting, external_id=None 2018-11-29 20:59:12,594 DEBUG nodepool.driver.static.StaticNodeProvider: Registered static node 35.221.20.172 2018-11-29 20:59:12,834 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 20:59:14,249 ERROR nodepool.NodeDeleter: Exception deleting instance 0f13ac4c-c10c-4892-8172-561e6f949d19 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 0f13ac4c-c10c-4892-8172-561e6f949d19 deletion 2018-11-29 20:59:14,379 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.544865369796753s 2018-11-29 20:59:14,395 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:15,117 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:15,423 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0261800289154053s 2018-11-29 20:59:15,538 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.4207627773284912s 2018-11-29 20:59:15,542 ERROR nodepool.NodeLauncher-0000318668: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318668: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 157, in _launchNode auto_ip=self.pool.auto_floating_ip) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 356, in waitForServer reuse=False, timeout=timeout) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6881, in wait_for_server nat_destination=nat_destination) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6898, in get_active_server "Error in creating the server", extra_data=dict(server=server)) openstack.exceptions.SDKException: Error in creating the server 2018-11-29 20:59:15,546 INFO nodepool.NodeLauncher-0000318668: Request 300-0000331925: Node 2a7089ab-753e-49c0-aaba-be29bdde925f scheduled for cleanup 2018-11-29 20:59:15,551 INFO nodepool.NodeLauncher-0000318668: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318668 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318668 2018-11-29 20:59:15,552 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 20:59:16,219 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 20:59:16,260 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:16,271 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 20:59:16,316 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 20:59:16,381 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 20:59:16,381 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 20:59:16,382 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 20:59:16,392 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318672 for request 300-0000331819 2018-11-29 20:59:16,395 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 20:59:16,401 INFO nodepool.NodeLauncher-0000318672: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318672 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318672 2018-11-29 20:59:17,174 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9138004779815674s 2018-11-29 20:59:17,253 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318671 (state: deleting, allocated_to: None) 2018-11-29 20:59:17,253 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 2a7089ab-753e-49c0-aaba-be29bdde925f from vexxhost-ansible-network-mtl1 2018-11-29 20:59:17,258 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 2a7089ab-753e-49c0-aaba-be29bdde925f 2018-11-29 20:59:17,300 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211293 (state: deleting, allocated_to: None) 2018-11-29 20:59:17,300 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 0f13ac4c-c10c-4892-8172-561e6f949d19 from rdo-cloud-tripleo 2018-11-29 20:59:17,305 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 0f13ac4c-c10c-4892-8172-561e6f949d19 2018-11-29 20:59:17,306 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 20:59:17,376 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.06994485855102539s 2018-11-29 20:59:17,437 ERROR nodepool.NodeDeleter: Exception deleting instance 52af0012-6653-4e5d-adcf-291841fe4c5f from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 52af0012-6653-4e5d-adcf-291841fe4c5f deletion 2018-11-29 20:59:19,195 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 3.6435394287109375s 2018-11-29 20:59:19,196 ERROR nodepool.NodeLauncher-0000318668: Request 300-0000331925: Launch attempt 2/3 failed for node 0000318668: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 20:59:19,196 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 1) 2018-11-29 20:59:19,196 INFO nodepool.NodeLauncher-0000318668: Quota exceeded, invalidating quota cache 2018-11-29 20:59:19,199 INFO nodepool.NodeLauncher-0000318668: Aborting node 0000318668 due to quota failure 2018-11-29 20:59:20,406 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 20:59:21,426 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 2.2295374870300293s 2018-11-29 20:59:21,427 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeDeleteServers (queue 0) 2018-11-29 20:59:21,427 ERROR nodepool.NodeLauncher-0000318672: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318672: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 20:59:21,431 INFO nodepool.NodeLauncher-0000318672: Quota exceeded, invalidating quota cache 2018-11-29 20:59:21,431 INFO nodepool.NodeLauncher-0000318672: Aborting node 0000318672 due to quota failure 2018-11-29 20:59:21,759 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3523352146148682s 2018-11-29 20:59:21,775 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:22,415 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:22,650 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8722851276397705s 2018-11-29 20:59:22,735 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 20:59:22,736 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 20:59:22,737 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 20:59:22,737 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 20:59:22,738 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 20:59:22,739 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 20:59:22,739 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 20:59:22,740 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 20:59:22,741 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 20:59:22,741 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 20:59:22,741 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 20:59:22,741 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 20:59:22,742 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 20:59:22,742 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 20:59:22,790 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211295 (state: deleting, allocated_to: None) 2018-11-29 20:59:22,790 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 52af0012-6653-4e5d-adcf-291841fe4c5f from rdo-cloud-tripleo 2018-11-29 20:59:22,795 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 52af0012-6653-4e5d-adcf-291841fe4c5f 2018-11-29 20:59:22,796 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 20:59:22,860 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.06401443481445312s 2018-11-29 20:59:23,140 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7245175838470459s 2018-11-29 20:59:24,025 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeDeleteServers in 2.5980727672576904s 2018-11-29 20:59:24,026 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:24,807 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.7814493179321289s 2018-11-29 20:59:24,810 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318671, state=deleting, external_id=2a7089ab-753e-49c0-aaba-be29bdde925f 2018-11-29 20:59:26,509 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 20:59:26,512 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 20:59:26,512 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 20:59:27,153 ERROR nodepool.NodeDeleter: Exception deleting instance e5999063-a1b8-412a-bd0e-ef26c58f0381 from rdo-cloud: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server e5999063-a1b8-412a-bd0e-ef26c58f0381 deletion 2018-11-29 20:59:27,786 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:27,952 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318668 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 20:59:27,953 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 20:59:27,956 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318668, state=deleting, external_id=None 2018-11-29 20:59:27,968 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318672 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 20:59:27,968 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 20:59:27,971 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318672, state=deleting, external_id=None 2018-11-29 20:59:28,159 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000260472 (state: deleting, allocated_to: None) 2018-11-29 20:59:28,160 INFO nodepool.DeletedNodeWorker: Deleting deleting instance e5999063-a1b8-412a-bd0e-ef26c58f0381 from rdo-cloud 2018-11-29 20:59:28,163 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:28,659 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8720948696136475s 2018-11-29 20:59:28,673 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 20:59:29,191 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.0278027057647705s 2018-11-29 20:59:29,198 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server e5999063-a1b8-412a-bd0e-ef26c58f0381 2018-11-29 20:59:29,199 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeDeleteServers (queue 0) 2018-11-29 20:59:29,280 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeDeleteServers in 0.0817863941192627s 2018-11-29 20:59:29,941 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2679443359375s 2018-11-29 20:59:33,603 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 20:59:33,604 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 20:59:33,606 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 20:59:33,607 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 20:59:33,608 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 20:59:33,609 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 20:59:33,612 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 20:59:33,612 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 20:59:33,612 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 20:59:33,612 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 20:59:33,612 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 20:59:33,613 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 20:59:33,613 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 20:59:33,614 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 20:59:33,805 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:34,328 ERROR nodepool.NodeDeleter: Exception deleting instance bbbdbcbc-cfa9-4ee1-828e-03a90bdda5ac from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server bbbdbcbc-cfa9-4ee1-828e-03a90bdda5ac deletion 2018-11-29 20:59:34,525 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.7189559936523438s 2018-11-29 20:59:35,212 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:35,909 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6967389583587646s 2018-11-29 20:59:35,971 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 20:59:36,517 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 20:59:37,239 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2661919593811035s 2018-11-29 20:59:37,434 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.9169430732727051s 2018-11-29 20:59:37,435 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 20:59:37,435 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 20:59:37,583 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1477978229522705s 2018-11-29 20:59:37,587 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:38,256 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6691646575927734s 2018-11-29 20:59:38,272 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 20:59:38,272 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 20:59:38,393 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 7, 'ram': 50176, 'instances': 9}} 2018-11-29 20:59:38,445 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 8}} 2018-11-29 20:59:38,445 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 20:59:38,445 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 20:59:38,455 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318673 for request 300-0000331925 2018-11-29 20:59:38,460 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 20:59:38,462 INFO nodepool.NodeLauncher-0000318673: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318673 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318673 2018-11-29 20:59:38,463 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 20:59:38,920 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000242421 (state: deleting, allocated_to: None) 2018-11-29 20:59:38,920 INFO nodepool.DeletedNodeWorker: Deleting deleting instance bbbdbcbc-cfa9-4ee1-828e-03a90bdda5ac from rdo-cloud-tripleo 2018-11-29 20:59:38,925 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server bbbdbcbc-cfa9-4ee1-828e-03a90bdda5ac 2018-11-29 20:59:38,926 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 20:59:38,991 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.06500840187072754s 2018-11-29 20:59:39,601 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.1377863883972168s 2018-11-29 20:59:39,601 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServers (queue 0) 2018-11-29 20:59:39,831 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServers in 0.22966384887695312s 2018-11-29 20:59:39,840 DEBUG nodepool.NodeLauncher-0000318673: Waiting for server 18ffac16-aa61-4683-adda-70db576be29d for node id: 0000318673 2018-11-29 20:59:39,929 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:41,102 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1725749969482422s 2018-11-29 20:59:41,300 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:42,024 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.723576545715332s 2018-11-29 20:59:43,271 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 20:59:44,322 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 20:59:44,328 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 20:59:44,330 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 20:59:44,330 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 20:59:44,331 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 20:59:44,331 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 20:59:44,331 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 20:59:44,332 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 20:59:44,335 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 20:59:44,336 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 20:59:44,340 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 20:59:44,340 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 20:59:44,342 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 20:59:44,344 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 20:59:44,494 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2227234840393066s 2018-11-29 20:59:44,844 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:45,152 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.30753350257873535s 2018-11-29 20:59:45,156 ERROR nodepool.NodeLauncher-0000318673: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318673: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 157, in _launchNode auto_ip=self.pool.auto_floating_ip) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 356, in waitForServer reuse=False, timeout=timeout) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6881, in wait_for_server nat_destination=nat_destination) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6898, in get_active_server "Error in creating the server", extra_data=dict(server=server)) openstack.exceptions.SDKException: Error in creating the server 2018-11-29 20:59:45,162 INFO nodepool.NodeLauncher-0000318673: Request 300-0000331925: Node 18ffac16-aa61-4683-adda-70db576be29d scheduled for cleanup 2018-11-29 20:59:45,167 INFO nodepool.NodeLauncher-0000318673: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318673 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318673 2018-11-29 20:59:45,167 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 20:59:46,060 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.8921749591827393s 2018-11-29 20:59:46,060 ERROR nodepool.NodeLauncher-0000318673: Request 300-0000331925: Launch attempt 2/3 failed for node 0000318673: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 20:59:46,060 INFO nodepool.NodeLauncher-0000318673: Quota exceeded, invalidating quota cache 2018-11-29 20:59:46,061 INFO nodepool.NodeLauncher-0000318673: Aborting node 0000318673 due to quota failure 2018-11-29 20:59:46,422 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:47,432 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0098297595977783s 2018-11-29 20:59:47,932 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:48,317 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:48,464 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 20:59:48,610 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.14514541625976562s 2018-11-29 20:59:48,610 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 20:59:48,611 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 20:59:48,710 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.3935892581939697s 2018-11-29 20:59:48,711 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-ad0c2270-ab9e-4c06-acf6-d68ea3ff8d56) 2018-11-29 20:59:48,818 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.20701885223388672s 2018-11-29 20:59:48,822 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 20:59:48,822 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 20:59:48,850 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 20:59:48,903 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 20:59:48,967 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 20:59:49,017 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 20:59:49,018 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 20:59:49,018 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 20:59:49,028 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318675 for request 300-0000331819 2018-11-29 20:59:49,034 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 20:59:49,034 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 20:59:49,036 INFO nodepool.NodeLauncher-0000318675: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318675 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318675 2018-11-29 20:59:49,036 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 20:59:49,160 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.227499008178711s 2018-11-29 20:59:49,551 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318674 (state: deleting, allocated_to: None) 2018-11-29 20:59:49,551 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 18ffac16-aa61-4683-adda-70db576be29d from vexxhost-ansible-network-mtl1 2018-11-29 20:59:49,558 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 18ffac16-aa61-4683-adda-70db576be29d 2018-11-29 20:59:49,563 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318673 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 20:59:49,563 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 20:59:49,568 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318673, state=deleting, external_id=None 2018-11-29 20:59:50,499 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4624223709106445s 2018-11-29 20:59:50,499 ERROR nodepool.NodeLauncher-0000318675: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318675: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 20:59:50,500 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeDeleteServers (queue 0) 2018-11-29 20:59:50,500 INFO nodepool.NodeLauncher-0000318675: Quota exceeded, invalidating quota cache 2018-11-29 20:59:50,503 INFO nodepool.NodeLauncher-0000318675: Aborting node 0000318675 due to quota failure 2018-11-29 20:59:50,523 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 20:59:51,851 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3284659385681152s 2018-11-29 20:59:51,973 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeDeleteServers in 1.472412109375s 2018-11-29 20:59:51,973 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:52,293 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.31999897956848145s 2018-11-29 20:59:52,295 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318674, state=deleting, external_id=18ffac16-aa61-4683-adda-70db576be29d 2018-11-29 20:59:52,528 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:53,796 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2683444023132324s 2018-11-29 20:59:55,077 INFO nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Assigning node request 2018-11-29 20:59:55,079 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4]: Declining node request 200-0000331926 because node type(s) [static-ansible] not available 2018-11-29 20:59:55,116 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Assigning node request 2018-11-29 20:59:55,116 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.large]: Declining node request 200-0000331927 because node type(s) [static-ansible] not available 2018-11-29 20:59:55,158 INFO nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Assigning node request 2018-11-29 20:59:55,159 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-rdo-ca-ymq-1-main]: Declining node request 200-0000331928 because node type(s) [static-ansible] not available 2018-11-29 20:59:55,175 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:55,203 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Assigning node request 2018-11-29 20:59:55,204 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.large]: Declining node request 200-0000331929 because node type(s) [static-ansible] not available 2018-11-29 20:59:55,244 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 20:59:55,245 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 200-0000331930 because node type(s) [static-ansible] not available 2018-11-29 20:59:55,297 INFO nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Assigning node request 2018-11-29 20:59:55,297 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-tripleo-ovb]: Declining node request 200-0000331931 because node type(s) [static-ansible] not available 2018-11-29 20:59:55,307 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 20:59:55,307 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 20:59:55,308 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 20:59:55,309 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 20:59:55,309 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 20:59:55,310 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 20:59:55,311 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 20:59:55,314 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 20:59:55,350 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Removing handler for request 200-0000331926 2018-11-29 20:59:55,351 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 20:59:55,359 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Removing handler for request 200-0000331927 2018-11-29 20:59:55,359 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 20:59:55,361 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Removing handler for request 200-0000331928 2018-11-29 20:59:55,361 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 20:59:55,364 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Removing handler for request 200-0000331929 2018-11-29 20:59:55,364 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 20:59:55,370 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 200-0000331930 2018-11-29 20:59:55,370 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 20:59:55,374 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Removing handler for request 200-0000331931 2018-11-29 20:59:55,374 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 20:59:56,265 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.0897481441497803s 2018-11-29 20:59:57,879 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 20:59:59,039 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 20:59:59,175 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.13504409790039062s 2018-11-29 20:59:59,175 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 20:59:59,177 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 20:59:59,181 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3016645908355713s 2018-11-29 20:59:59,192 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:59,409 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.2319965362548828s 2018-11-29 20:59:59,413 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 20:59:59,811 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.39804720878601074s 2018-11-29 20:59:59,829 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 20:59:59,829 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 20:59:59,940 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 20:59:59,998 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 20:59:59,998 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 20:59:59,998 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:00:00,021 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318676 for request 300-0000331925 2018-11-29 21:00:00,028 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:00:00,028 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:00:00,029 INFO nodepool.NodeLauncher-0000318676: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318676 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318676 2018-11-29 21:00:00,030 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:00:00,522 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318675 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:00:00,522 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:00:00,525 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318675, state=deleting, external_id=None 2018-11-29 21:00:00,793 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.595987319946289s 2018-11-29 21:00:01,353 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.3233702182769775s 2018-11-29 21:00:01,354 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServers (queue 0) 2018-11-29 21:00:01,756 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServers in 0.4018585681915283s 2018-11-29 21:00:01,764 DEBUG nodepool.NodeLauncher-0000318676: Waiting for server c5fb2040-7749-4654-b97a-dca09cb3f724 for node id: 0000318676 2018-11-29 21:00:02,060 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:02,923 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8625996112823486s 2018-11-29 21:00:05,216 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:00:05,886 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:00:05,887 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Declining node request 200-0000331926 because node type(s) [static-ansible] not available 2018-11-29 21:00:05,910 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:00:05,910 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Declining node request 200-0000331927 because node type(s) [static-ansible] not available 2018-11-29 21:00:05,941 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Assigning node request 2018-11-29 21:00:05,941 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.small]: Declining node request 200-0000331928 because node type(s) [static-ansible] not available 2018-11-29 21:00:05,975 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Assigning node request 2018-11-29 21:00:05,975 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1]: Declining node request 200-0000331929 because node type(s) [static-ansible] not available 2018-11-29 21:00:06,007 INFO nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Assigning node request 2018-11-29 21:00:06,007 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Declining node request 200-0000331930 because node type(s) [static-ansible] not available 2018-11-29 21:00:06,008 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Assigning node request 2018-11-29 21:00:06,009 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.large]: Declining node request 200-0000331926 because node type(s) [static-ansible] not available 2018-11-29 21:00:06,041 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Assigning node request 2018-11-29 21:00:06,043 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.large]: Declining node request 200-0000331927 because node type(s) [static-ansible] not available 2018-11-29 21:00:06,043 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Assigning node request 2018-11-29 21:00:06,045 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.small]: Declining node request 200-0000331931 because node type(s) [static-ansible] not available 2018-11-29 21:00:06,054 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:00:06,077 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:00:06,093 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 21:00:06,093 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 200-0000331928 because node type(s) [static-ansible] not available 2018-11-29 21:00:06,129 INFO nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Assigning node request 2018-11-29 21:00:06,130 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-tripleo-ovb]: Declining node request 200-0000331929 because node type(s) [static-ansible] not available 2018-11-29 21:00:06,138 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 200-0000331927 2018-11-29 21:00:06,138 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:00:06,139 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 200-0000331926 2018-11-29 21:00:06,139 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Removing handler for request 200-0000331928 2018-11-29 21:00:06,140 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:00:06,140 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:00:06,140 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:00:06,146 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Removing handler for request 200-0000331929 2018-11-29 21:00:06,147 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:00:06,148 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Removing handler for request 200-0000331930 2018-11-29 21:00:06,148 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:00:06,150 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:00:06,152 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Removing handler for request 200-0000331931 2018-11-29 21:00:06,152 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:00:06,175 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Assigning node request 2018-11-29 21:00:06,176 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.large]: Declining node request 200-0000331930 because node type(s) [static-ansible] not available 2018-11-29 21:00:06,194 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Assigning node request 2018-11-29 21:00:06,196 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.large]: Declining node request 200-0000331931 because node type(s) [static-ansible] not available 2018-11-29 21:00:06,208 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 200-0000331928 2018-11-29 21:00:06,208 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:00:06,209 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Removing handler for request 200-0000331929 2018-11-29 21:00:06,209 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:00:06,234 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Removing handler for request 200-0000331926 2018-11-29 21:00:06,234 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Removing handler for request 200-0000331930 2018-11-29 21:00:06,234 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:00:06,243 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Removing handler for request 200-0000331927 2018-11-29 21:00:06,243 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Removing handler for request 200-0000331931 2018-11-29 21:00:06,243 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:00:06,445 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.228468656539917s 2018-11-29 21:00:06,464 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:06,766 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:07,688 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.9217624664306641s 2018-11-29 21:00:07,693 ERROR nodepool.NodeLauncher-0000318676: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318676: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 157, in _launchNode auto_ip=self.pool.auto_floating_ip) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 356, in waitForServer reuse=False, timeout=timeout) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6881, in wait_for_server nat_destination=nat_destination) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6898, in get_active_server "Error in creating the server", extra_data=dict(server=server)) openstack.exceptions.SDKException: Error in creating the server 2018-11-29 21:00:07,698 INFO nodepool.NodeLauncher-0000318676: Request 300-0000331925: Node c5fb2040-7749-4654-b97a-dca09cb3f724 scheduled for cleanup 2018-11-29 21:00:07,703 INFO nodepool.NodeLauncher-0000318676: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318676 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318676 2018-11-29 21:00:07,704 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:00:07,732 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.26796293258667s 2018-11-29 21:00:08,290 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:08,996 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7062129974365234s 2018-11-29 21:00:09,135 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4315643310546875s 2018-11-29 21:00:09,136 ERROR nodepool.NodeLauncher-0000318676: Request 300-0000331925: Launch attempt 2/3 failed for node 0000318676: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:00:09,136 INFO nodepool.NodeLauncher-0000318676: Quota exceeded, invalidating quota cache 2018-11-29 21:00:09,136 INFO nodepool.NodeLauncher-0000318676: Aborting node 0000318676 due to quota failure 2018-11-29 21:00:10,035 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:00:10,271 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.23619532585144043s 2018-11-29 21:00:10,272 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:00:10,272 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:00:10,410 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1376957893371582s 2018-11-29 21:00:10,414 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:00:10,415 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:00:10,475 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:00:10,522 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:00:10,573 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:00:10,573 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:00:10,573 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:00:10,593 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318678 for request 300-0000331819 2018-11-29 21:00:10,601 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:00:10,601 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:00:10,602 INFO nodepool.NodeLauncher-0000318678: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318678 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318678 2018-11-29 21:00:10,603 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:00:11,358 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318676 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:00:11,358 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:00:11,362 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318676, state=deleting, external_id=None 2018-11-29 21:00:11,387 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318677 (state: deleting, allocated_to: None) 2018-11-29 21:00:11,387 INFO nodepool.DeletedNodeWorker: Deleting deleting instance c5fb2040-7749-4654-b97a-dca09cb3f724 from vexxhost-ansible-network-mtl1 2018-11-29 21:00:11,394 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server c5fb2040-7749-4654-b97a-dca09cb3f724 2018-11-29 21:00:12,493 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:00:13,244 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 2.640590190887451s 2018-11-29 21:00:13,245 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeDeleteServers (queue 0) 2018-11-29 21:00:13,244 ERROR nodepool.NodeLauncher-0000318678: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318678: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:00:13,249 INFO nodepool.NodeLauncher-0000318678: Quota exceeded, invalidating quota cache 2018-11-29 21:00:13,249 INFO nodepool.NodeLauncher-0000318678: Aborting node 0000318678 due to quota failure 2018-11-29 21:00:13,808 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3143665790557861s 2018-11-29 21:00:13,818 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:14,592 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.7686347961425781s 2018-11-29 21:00:14,947 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:15,476 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeDeleteServers in 2.2314138412475586s 2018-11-29 21:00:15,477 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:15,650 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7029948234558105s 2018-11-29 21:00:16,145 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:00:16,147 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Accepting node request 200-0000331926 2018-11-29 21:00:16,177 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6999719142913818s 2018-11-29 21:00:16,180 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318677, state=deleting, external_id=c5fb2040-7749-4654-b97a-dca09cb3f724 2018-11-29 21:00:16,356 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:00:16,357 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 200-0000331927 because node type(s) [static-ansible] not available 2018-11-29 21:00:16,470 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:00:16,471 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 200-0000331928 because node type(s) [static-ansible] not available 2018-11-29 21:00:16,576 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:00:16,576 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 200-0000331929 because node type(s) [static-ansible] not available 2018-11-29 21:00:16,685 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:00:16,685 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 200-0000331930 because node type(s) [static-ansible] not available 2018-11-29 21:00:16,804 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:00:16,804 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 200-0000331931 because node type(s) [static-ansible] not available 2018-11-29 21:00:16,806 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Assigning node request 2018-11-29 21:00:16,807 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.small]: Declining node request 200-0000331927 because node type(s) [static-ansible] not available 2018-11-29 21:00:16,819 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:00:16,820 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Declining node request 200-0000331928 because node type(s) [static-ansible] not available 2018-11-29 21:00:16,859 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Locked existing node 0000318664 for request 200-0000331926 2018-11-29 21:00:16,881 INFO nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Assigning node request 2018-11-29 21:00:16,881 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-rdo-ca-ymq-1-main]: Declining node request 200-0000331929 because node type(s) [static-ansible] not available 2018-11-29 21:00:16,903 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Assigning node request 2018-11-29 21:00:16,903 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1]: Declining node request 200-0000331930 because node type(s) [static-ansible] not available 2018-11-29 21:00:16,923 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 200-0000331927 2018-11-29 21:00:16,923 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 200-0000331928 2018-11-29 21:00:16,923 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 200-0000331929 2018-11-29 21:00:16,924 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 200-0000331930 2018-11-29 21:00:16,924 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 200-0000331931 2018-11-29 21:00:16,924 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:00:16,936 INFO nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Assigning node request 2018-11-29 21:00:16,936 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Declining node request 200-0000331931 because node type(s) [static-ansible] not available 2018-11-29 21:00:16,937 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 21:00:16,938 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 200-0000331927 because node type(s) [static-ansible] not available 2018-11-29 21:00:16,953 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:00:16,957 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:00:16,960 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:00:16,975 INFO nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Assigning node request 2018-11-29 21:00:16,978 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-tripleo-ovb]: Declining node request 200-0000331928 because node type(s) [static-ansible] not available 2018-11-29 21:00:17,000 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Assigning node request 2018-11-29 21:00:17,000 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.large]: Declining node request 200-0000331929 because node type(s) [static-ansible] not available 2018-11-29 21:00:17,005 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Removing handler for request 200-0000331927 2018-11-29 21:00:17,005 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:00:17,008 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 200-0000331928 2018-11-29 21:00:17,009 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:00:17,011 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Fulfilled node request 200-0000331926 2018-11-29 21:00:17,017 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unlocked node 0000318664 for request 200-0000331926 2018-11-29 21:00:17,023 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Assigning node request 2018-11-29 21:00:17,024 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.large]: Declining node request 200-0000331930 because node type(s) [static-ansible] not available 2018-11-29 21:00:17,060 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Removing handler for request 200-0000331929 2018-11-29 21:00:17,061 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:00:17,061 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Removing handler for request 200-0000331930 2018-11-29 21:00:17,061 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:00:17,061 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Removing handler for request 200-0000331931 2018-11-29 21:00:17,061 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:00:17,062 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 200-0000331926 2018-11-29 21:00:17,062 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:00:17,219 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 21:00:17,220 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 200-0000331931 because node type(s) [static-ansible] not available 2018-11-29 21:00:17,236 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Removing handler for request 200-0000331928 2018-11-29 21:00:17,237 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:00:17,245 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Removing handler for request 200-0000331929 2018-11-29 21:00:17,245 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:00:17,245 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Removing handler for request 200-0000331930 2018-11-29 21:00:17,245 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:00:17,269 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 200-0000331927 2018-11-29 21:00:17,269 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 200-0000331931 2018-11-29 21:00:17,269 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:00:19,767 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:20,607 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:00:20,825 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0571784973144531s 2018-11-29 21:00:20,838 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:00:20,881 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2738766670227051s 2018-11-29 21:00:20,881 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:00:20,881 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:00:21,013 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1314868927001953s 2018-11-29 21:00:21,016 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:00:21,016 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:00:21,022 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:21,126 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:00:21,174 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:00:21,174 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:00:21,174 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:00:21,205 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318679 for request 300-0000331925 2018-11-29 21:00:21,211 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:00:21,212 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:00:21,213 INFO nodepool.NodeLauncher-0000318679: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318679 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318679 2018-11-29 21:00:21,214 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:00:22,113 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.0905578136444092s 2018-11-29 21:00:22,314 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4760792255401611s 2018-11-29 21:00:22,425 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318678 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:00:22,425 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:00:22,429 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318678, state=deleting, external_id=None 2018-11-29 21:00:23,303 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 2.0894417762756348s 2018-11-29 21:00:23,304 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServers (queue 0) 2018-11-29 21:00:23,629 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServers in 0.32506895065307617s 2018-11-29 21:00:23,635 DEBUG nodepool.NodeLauncher-0000318679: Waiting for server 1032a9c5-e952-4962-948a-e3181532c570 for node id: 0000318679 2018-11-29 21:00:23,636 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:24,001 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.3646354675292969s 2018-11-29 21:00:24,335 ERROR nodepool.NodeDeleter: Exception deleting instance fbf7b562-5826-4b67-a10b-87b962a17f12 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server fbf7b562-5826-4b67-a10b-87b962a17f12 deletion 2018-11-29 21:00:26,059 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:27,133 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:00:27,300 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:00:27,301 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Declining node request 200-0000331927 because node type(s) [static-ansible] not available 2018-11-29 21:00:27,332 INFO nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Assigning node request 2018-11-29 21:00:27,333 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4]: Declining node request 200-0000331928 because node type(s) [static-ansible] not available 2018-11-29 21:00:27,366 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.30704665184021s 2018-11-29 21:00:27,393 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Assigning node request 2018-11-29 21:00:27,394 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.small]: Declining node request 200-0000331929 because node type(s) [static-ansible] not available 2018-11-29 21:00:27,486 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:00:27,486 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Declining node request 200-0000331930 because node type(s) [static-ansible] not available 2018-11-29 21:00:27,528 INFO nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Assigning node request 2018-11-29 21:00:27,528 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4]: Declining node request 200-0000331931 because node type(s) [static-ansible] not available 2018-11-29 21:00:27,574 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Assigning node request 2018-11-29 21:00:27,574 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.small]: Declining node request 200-0000331929 because node type(s) [static-ansible] not available 2018-11-29 21:00:27,583 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Removing handler for request 200-0000331929 2018-11-29 21:00:27,584 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:00:27,674 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:00:27,675 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:27,680 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 200-0000331927 2018-11-29 21:00:27,681 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 200-0000331930 2018-11-29 21:00:27,681 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:00:27,700 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Removing handler for request 200-0000331928 2018-11-29 21:00:27,701 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Removing handler for request 200-0000331931 2018-11-29 21:00:27,701 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:00:27,715 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Assigning node request 2018-11-29 21:00:27,715 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.small]: Declining node request 200-0000331930 because node type(s) [static-ansible] not available 2018-11-29 21:00:27,726 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:00:27,727 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Accepting node request 200-0000331927 2018-11-29 21:00:27,750 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Assigning node request 2018-11-29 21:00:27,751 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1]: Declining node request 200-0000331928 because node type(s) [static-ansible] not available 2018-11-29 21:00:27,774 INFO nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Assigning node request 2018-11-29 21:00:27,775 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Declining node request 200-0000331929 because node type(s) [static-ansible] not available 2018-11-29 21:00:27,776 INFO nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Assigning node request 2018-11-29 21:00:27,776 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-rdo-ca-ymq-1-main]: Declining node request 200-0000331931 because node type(s) [static-ansible] not available 2018-11-29 21:00:27,805 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Removing handler for request 200-0000331929 2018-11-29 21:00:27,806 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Removing handler for request 200-0000331930 2018-11-29 21:00:27,806 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:00:27,842 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Removing handler for request 200-0000331931 2018-11-29 21:00:27,842 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:00:27,843 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Removing handler for request 200-0000331928 2018-11-29 21:00:27,844 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:00:27,847 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Removing handler for request 200-0000331929 2018-11-29 21:00:27,847 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:00:27,912 INFO nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Assigning node request 2018-11-29 21:00:27,913 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-tripleo-ovb]: Declining node request 200-0000331930 because node type(s) [static-ansible] not available 2018-11-29 21:00:27,914 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Assigning node request 2018-11-29 21:00:27,915 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.large]: Declining node request 200-0000331928 because node type(s) [static-ansible] not available 2018-11-29 21:00:27,929 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:00:27,930 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 21:00:27,930 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 200-0000331929 because node type(s) [static-ansible] not available 2018-11-29 21:00:27,985 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211298 (state: deleting, allocated_to: None) 2018-11-29 21:00:27,985 INFO nodepool.DeletedNodeWorker: Deleting deleting instance fbf7b562-5826-4b67-a10b-87b962a17f12 from rdo-cloud-tripleo 2018-11-29 21:00:28,002 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:00:28,020 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Removing handler for request 200-0000331930 2018-11-29 21:00:28,021 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:00:28,034 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Assigning node request 2018-11-29 21:00:28,035 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.large]: Declining node request 200-0000331931 because node type(s) [static-ansible] not available 2018-11-29 21:00:28,036 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 200-0000331929 2018-11-29 21:00:28,037 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:00:28,067 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Removing handler for request 200-0000331928 2018-11-29 21:00:28,067 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Removing handler for request 200-0000331931 2018-11-29 21:00:28,067 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:00:28,165 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Locked existing node 0000318669 for request 200-0000331927 2018-11-29 21:00:28,176 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:00:28,176 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Accepting node request 200-0000331928 2018-11-29 21:00:28,293 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Locked existing node 0000318670 for request 200-0000331928 2018-11-29 21:00:28,300 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:00:28,301 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Accepting node request 200-0000331929 2018-11-29 21:00:28,447 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Locked existing node 0000318644 for request 200-0000331929 2018-11-29 21:00:28,462 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:00:28,462 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Accepting node request 200-0000331930 2018-11-29 21:00:28,584 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Locked existing node 0000318639 for request 200-0000331930 2018-11-29 21:00:28,593 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:00:28,593 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Accepting node request 200-0000331931 2018-11-29 21:00:28,715 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Locked existing node 0000318651 for request 200-0000331931 2018-11-29 21:00:28,720 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.0449168682098389s 2018-11-29 21:00:28,727 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Fulfilled node request 200-0000331927 2018-11-29 21:00:28,730 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unlocked node 0000318669 for request 200-0000331927 2018-11-29 21:00:28,735 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 200-0000331927 2018-11-29 21:00:28,736 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Fulfilled node request 200-0000331928 2018-11-29 21:00:28,738 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unlocked node 0000318670 for request 200-0000331928 2018-11-29 21:00:28,747 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 200-0000331928 2018-11-29 21:00:28,749 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Fulfilled node request 200-0000331929 2018-11-29 21:00:28,752 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unlocked node 0000318644 for request 200-0000331929 2018-11-29 21:00:28,761 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 200-0000331929 2018-11-29 21:00:28,762 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Fulfilled node request 200-0000331930 2018-11-29 21:00:28,765 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unlocked node 0000318639 for request 200-0000331930 2018-11-29 21:00:28,771 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 200-0000331930 2018-11-29 21:00:28,772 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Fulfilled node request 200-0000331931 2018-11-29 21:00:28,774 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unlocked node 0000318651 for request 200-0000331931 2018-11-29 21:00:28,781 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 200-0000331931 2018-11-29 21:00:28,781 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:00:29,008 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:29,328 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.31969642639160156s 2018-11-29 21:00:29,334 ERROR nodepool.NodeLauncher-0000318679: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318679: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 157, in _launchNode auto_ip=self.pool.auto_floating_ip) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 356, in waitForServer reuse=False, timeout=timeout) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6881, in wait_for_server nat_destination=nat_destination) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6898, in get_active_server "Error in creating the server", extra_data=dict(server=server)) openstack.exceptions.SDKException: Error in creating the server 2018-11-29 21:00:29,338 INFO nodepool.NodeLauncher-0000318679: Request 300-0000331925: Node 1032a9c5-e952-4962-948a-e3181532c570 scheduled for cleanup 2018-11-29 21:00:29,344 INFO nodepool.NodeLauncher-0000318679: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318679 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318679 2018-11-29 21:00:29,344 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:00:29,588 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.585240125656128s 2018-11-29 21:00:29,608 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server fbf7b562-5826-4b67-a10b-87b962a17f12 2018-11-29 21:00:29,608 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:00:29,675 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.06662392616271973s 2018-11-29 21:00:30,620 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2759160995483398s 2018-11-29 21:00:30,621 ERROR nodepool.NodeLauncher-0000318679: Request 300-0000331925: Launch attempt 2/3 failed for node 0000318679: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:00:30,621 INFO nodepool.NodeLauncher-0000318679: Quota exceeded, invalidating quota cache 2018-11-29 21:00:30,621 INFO nodepool.NodeLauncher-0000318679: Aborting node 0000318679 due to quota failure 2018-11-29 21:00:31,216 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:00:31,354 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.13782477378845215s 2018-11-29 21:00:31,355 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:00:31,355 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:00:31,569 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.21358609199523926s 2018-11-29 21:00:31,572 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:00:31,572 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:00:31,625 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:00:31,676 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:00:31,740 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:00:31,740 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:00:31,740 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:00:31,749 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318681 for request 300-0000331819 2018-11-29 21:00:31,756 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:00:31,756 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:00:31,758 INFO nodepool.NodeLauncher-0000318681: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318681 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318681 2018-11-29 21:00:31,758 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:00:32,508 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:33,154 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.3956193923950195s 2018-11-29 21:00:33,154 ERROR nodepool.NodeLauncher-0000318681: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318681: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:00:33,155 INFO nodepool.NodeLauncher-0000318681: Quota exceeded, invalidating quota cache 2018-11-29 21:00:33,155 INFO nodepool.NodeLauncher-0000318681: Aborting node 0000318681 due to quota failure 2018-11-29 21:00:33,195 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318680 (state: deleting, allocated_to: None) 2018-11-29 21:00:33,195 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 1032a9c5-e952-4962-948a-e3181532c570 from vexxhost-ansible-network-mtl1 2018-11-29 21:00:33,203 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 1032a9c5-e952-4962-948a-e3181532c570 2018-11-29 21:00:33,203 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeDeleteServers (queue 0) 2018-11-29 21:00:33,241 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318679 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:00:33,242 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:00:33,245 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318679, state=deleting, external_id=None 2018-11-29 21:00:33,547 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.03853440284729s 2018-11-29 21:00:34,121 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeDeleteServers in 0.9175496101379395s 2018-11-29 21:00:34,135 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:34,912 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.776437520980835s 2018-11-29 21:00:35,708 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:00:36,125 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:36,619 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.49344873428344727s 2018-11-29 21:00:36,621 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318680, state=deleting, external_id=1032a9c5-e952-4962-948a-e3181532c570 2018-11-29 21:00:37,163 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4500291347503662s 2018-11-29 21:00:37,231 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:00:37,657 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:00:37,806 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:00:37,822 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:00:37,844 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:00:37,962 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:00:38,006 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:00:38,008 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:00:38,010 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:00:38,056 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:00:38,135 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:00:38,147 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:00:38,169 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:00:38,641 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:38,876 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:00:39,879 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.238105058670044s 2018-11-29 21:00:40,741 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:41,588 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8466522693634033s 2018-11-29 21:00:41,761 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:00:41,973 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2117900848388672s 2018-11-29 21:00:41,973 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:00:41,974 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:00:42,102 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1281435489654541s 2018-11-29 21:00:42,105 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:42,707 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6013321876525879s 2018-11-29 21:00:42,722 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:00:42,722 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:00:42,836 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:00:42,886 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:00:42,887 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:00:42,887 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:00:42,896 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318682 for request 300-0000331925 2018-11-29 21:00:42,904 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:00:42,904 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:00:42,906 INFO nodepool.NodeLauncher-0000318682: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318682 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318682 2018-11-29 21:00:42,906 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:00:43,190 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:00:43,841 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318681 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:00:43,841 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:00:43,844 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318681, state=deleting, external_id=None 2018-11-29 21:00:44,672 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.7657594680786133s 2018-11-29 21:00:44,673 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServers (queue 0) 2018-11-29 21:00:44,678 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4880144596099854s 2018-11-29 21:00:45,095 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServers in 0.4218158721923828s 2018-11-29 21:00:45,100 DEBUG nodepool.NodeLauncher-0000318682: Waiting for server 4ea8d4e7-0ec5-474a-b30a-4c358dca1d08 for node id: 0000318682 2018-11-29 21:00:45,109 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:46,523 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.4135949611663818s 2018-11-29 21:00:46,937 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:47,298 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:00:47,717 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:00:47,851 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9140951633453369s 2018-11-29 21:00:47,923 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:00:47,953 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:00:47,983 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:00:48,087 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:00:48,190 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:00:48,194 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:00:48,194 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:00:48,251 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:00:48,295 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:00:48,302 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:00:48,307 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:00:48,953 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:00:49,279 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:49,664 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.38562440872192383s 2018-11-29 21:00:49,665 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f10b842e-8458-485d-a47c-8304ddef2952) 2018-11-29 21:00:49,815 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:00:50,103 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:50,676 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5719265937805176s 2018-11-29 21:00:50,680 ERROR nodepool.NodeLauncher-0000318682: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318682: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 157, in _launchNode auto_ip=self.pool.auto_floating_ip) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 356, in waitForServer reuse=False, timeout=timeout) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6881, in wait_for_server nat_destination=nat_destination) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6898, in get_active_server "Error in creating the server", extra_data=dict(server=server)) openstack.exceptions.SDKException: Error in creating the server 2018-11-29 21:00:50,685 INFO nodepool.NodeLauncher-0000318682: Request 300-0000331925: Node 4ea8d4e7-0ec5-474a-b30a-4c358dca1d08 scheduled for cleanup 2018-11-29 21:00:50,690 INFO nodepool.NodeLauncher-0000318682: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318682 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318682 2018-11-29 21:00:50,691 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:00:50,710 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:00:51,899 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2076380252838135s 2018-11-29 21:00:51,899 ERROR nodepool.NodeLauncher-0000318682: Request 300-0000331925: Launch attempt 2/3 failed for node 0000318682: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:00:51,900 INFO nodepool.NodeLauncher-0000318682: Quota exceeded, invalidating quota cache 2018-11-29 21:00:51,900 INFO nodepool.NodeLauncher-0000318682: Aborting node 0000318682 due to quota failure 2018-11-29 21:00:52,108 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.397444248199463s 2018-11-29 21:00:52,123 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:52,909 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:00:53,100 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.19101428985595703s 2018-11-29 21:00:53,101 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:00:53,101 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:00:53,243 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.14177870750427246s 2018-11-29 21:00:53,246 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:00:53,247 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:00:53,307 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:00:53,356 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:00:53,406 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:00:53,406 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:00:53,407 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:00:53,416 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318684 for request 300-0000331819 2018-11-29 21:00:53,425 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:00:53,425 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:00:53,426 INFO nodepool.NodeLauncher-0000318684: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318684 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318684 2018-11-29 21:00:53,427 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:00:53,613 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:53,631 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.507568597793579s 2018-11-29 21:00:54,236 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6232233047485352s 2018-11-29 21:00:54,650 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2231979370117188s 2018-11-29 21:00:54,650 ERROR nodepool.NodeLauncher-0000318684: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318684: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:00:54,651 INFO nodepool.NodeLauncher-0000318684: Quota exceeded, invalidating quota cache 2018-11-29 21:00:54,651 INFO nodepool.NodeLauncher-0000318684: Aborting node 0000318684 due to quota failure 2018-11-29 21:00:54,695 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318683 (state: deleting, allocated_to: None) 2018-11-29 21:00:54,695 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 4ea8d4e7-0ec5-474a-b30a-4c358dca1d08 from vexxhost-ansible-network-mtl1 2018-11-29 21:00:54,700 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 4ea8d4e7-0ec5-474a-b30a-4c358dca1d08 2018-11-29 21:00:54,701 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeDeleteServers (queue 0) 2018-11-29 21:00:54,710 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318682 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:00:54,710 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:00:54,714 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318682, state=deleting, external_id=None 2018-11-29 21:00:55,805 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeDeleteServers in 1.104081630706787s 2018-11-29 21:00:55,806 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:56,355 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5497119426727295s 2018-11-29 21:00:56,358 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318683, state=deleting, external_id=4ea8d4e7-0ec5-474a-b30a-4c358dca1d08 2018-11-29 21:00:57,374 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:00:57,781 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:00:58,043 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:00:58,102 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:00:58,136 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:00:58,141 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:00:58,188 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:00:58,348 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:00:58,350 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:00:58,355 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:00:58,446 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:00:58,481 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:00:58,484 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:00:58,486 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:00:59,038 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:00:59,579 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4378042221069336s 2018-11-29 21:00:59,594 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:00:59,878 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:00,497 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6188852787017822s 2018-11-29 21:01:00,819 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2226252555847168s 2018-11-29 21:01:03,430 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:01:03,618 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.1874403953552246s 2018-11-29 21:01:03,618 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:01:03,619 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:01:03,825 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.20575261116027832s 2018-11-29 21:01:03,828 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:04,665 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.8361051082611084s 2018-11-29 21:01:04,681 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:01:04,681 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:01:04,785 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:01:04,847 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:01:04,848 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:01:04,848 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:01:04,857 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318685 for request 300-0000331925 2018-11-29 21:01:04,864 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:01:04,864 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:01:04,866 INFO nodepool.NodeLauncher-0000318685: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318685 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318685 2018-11-29 21:01:04,867 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:01:05,242 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318684 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:01:05,242 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:01:05,246 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318684, state=deleting, external_id=None 2018-11-29 21:01:05,607 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:01:06,265 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:06,376 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.5092504024505615s 2018-11-29 21:01:06,377 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServers (queue 0) 2018-11-29 21:01:06,874 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServers in 0.497631311416626s 2018-11-29 21:01:06,883 DEBUG nodepool.NodeLauncher-0000318685: Waiting for server 8c8e2d0b-58dd-4cb8-a764-1fabb5592420 for node id: 0000318685 2018-11-29 21:01:07,138 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.53029203414917s 2018-11-29 21:01:07,153 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:07,440 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:01:07,563 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.2980117797851562s 2018-11-29 21:01:07,843 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:01:08,120 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:01:08,192 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0388154983520508s 2018-11-29 21:01:08,227 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:01:08,273 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:01:08,303 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:01:08,464 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:01:08,466 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:01:08,472 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:01:08,586 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:01:08,629 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:01:08,632 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:01:08,633 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:01:09,129 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:01:09,130 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Accepting node request 100-0000331932 2018-11-29 21:01:09,279 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Locked existing node 0000318109 for request 100-0000331932 2018-11-29 21:01:09,291 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:01:09,291 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Accepting node request 100-0000331933 2018-11-29 21:01:09,365 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331933 2018-11-29 21:01:09,365 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Pausing request handling to satisfy request 100-0000331933 2018-11-29 21:01:09,436 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Fulfilled node request 100-0000331932 2018-11-29 21:01:09,439 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unlocked node 0000318109 for request 100-0000331932 2018-11-29 21:01:09,445 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 100-0000331932 2018-11-29 21:01:09,445 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331933'] 2018-11-29 21:01:11,888 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:12,232 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.3434422016143799s 2018-11-29 21:01:12,240 ERROR nodepool.NodeLauncher-0000318685: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318685: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 157, in _launchNode auto_ip=self.pool.auto_floating_ip) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 356, in waitForServer reuse=False, timeout=timeout) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6881, in wait_for_server nat_destination=nat_destination) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6898, in get_active_server "Error in creating the server", extra_data=dict(server=server)) openstack.exceptions.SDKException: Error in creating the server 2018-11-29 21:01:12,247 INFO nodepool.NodeLauncher-0000318685: Request 300-0000331925: Node 8c8e2d0b-58dd-4cb8-a764-1fabb5592420 scheduled for cleanup 2018-11-29 21:01:12,254 INFO nodepool.NodeLauncher-0000318685: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318685 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318685 2018-11-29 21:01:12,255 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:01:13,169 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:01:13,579 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:13,805 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.549807071685791s 2018-11-29 21:01:13,805 ERROR nodepool.NodeLauncher-0000318685: Request 300-0000331925: Launch attempt 2/3 failed for node 0000318685: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:01:13,806 INFO nodepool.NodeLauncher-0000318685: Quota exceeded, invalidating quota cache 2018-11-29 21:01:13,806 INFO nodepool.NodeLauncher-0000318685: Aborting node 0000318685 due to quota failure 2018-11-29 21:01:14,347 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1773583889007568s 2018-11-29 21:01:14,362 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:14,713 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.1333446502685547s 2018-11-29 21:01:14,869 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:01:15,007 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.13811945915222168s 2018-11-29 21:01:15,008 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:01:15,008 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:01:15,202 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.193864107131958s 2018-11-29 21:01:15,205 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:01:15,206 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:01:15,265 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:01:15,316 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:01:15,330 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9649291038513184s 2018-11-29 21:01:15,390 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:01:15,390 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:01:15,390 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:01:15,400 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318687 for request 300-0000331819 2018-11-29 21:01:15,408 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:01:15,408 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:01:15,410 INFO nodepool.NodeLauncher-0000318687: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318687 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318687 2018-11-29 21:01:15,410 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:01:15,778 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318685 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:01:15,778 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:01:15,782 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318685, state=deleting, external_id=None 2018-11-29 21:01:15,803 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318686 (state: deleting, allocated_to: None) 2018-11-29 21:01:15,803 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 8c8e2d0b-58dd-4cb8-a764-1fabb5592420 from vexxhost-ansible-network-mtl1 2018-11-29 21:01:15,809 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 8c8e2d0b-58dd-4cb8-a764-1fabb5592420 2018-11-29 21:01:16,489 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.0786144733428955s 2018-11-29 21:01:16,500 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeDeleteServers (queue 0) 2018-11-29 21:01:16,489 ERROR nodepool.NodeLauncher-0000318687: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318687: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:01:16,526 INFO nodepool.NodeLauncher-0000318687: Quota exceeded, invalidating quota cache 2018-11-29 21:01:16,533 INFO nodepool.NodeLauncher-0000318687: Aborting node 0000318687 due to quota failure 2018-11-29 21:01:17,214 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeDeleteServers in 0.7060627937316895s 2018-11-29 21:01:17,522 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:01:17,522 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 100-0000331934 because node type(s) [static-ansible] not available 2018-11-29 21:01:17,592 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:01:17,593 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 100-0000331935 because node type(s) [static-ansible] not available 2018-11-29 21:01:17,615 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:01:17,615 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 100-0000331936 because node type(s) [static-ansible] not available 2018-11-29 21:01:17,635 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:01:17,635 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 100-0000331937 because node type(s) [static-ansible] not available 2018-11-29 21:01:17,650 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 100-0000331934 2018-11-29 21:01:17,650 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 100-0000331935 2018-11-29 21:01:17,650 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 100-0000331936 2018-11-29 21:01:17,650 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 100-0000331937 2018-11-29 21:01:17,651 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:01:17,925 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Assigning node request 2018-11-29 21:01:17,926 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.small]: Declining node request 100-0000331934 because node type(s) [static-ansible] not available 2018-11-29 21:01:17,945 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Assigning node request 2018-11-29 21:01:17,945 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.small]: Declining node request 100-0000331935 because node type(s) [static-ansible] not available 2018-11-29 21:01:17,965 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Assigning node request 2018-11-29 21:01:17,965 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.small]: Declining node request 100-0000331936 because node type(s) [static-ansible] not available 2018-11-29 21:01:17,983 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Assigning node request 2018-11-29 21:01:17,983 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.small]: Declining node request 100-0000331937 because node type(s) [static-ansible] not available 2018-11-29 21:01:17,997 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Removing handler for request 100-0000331934 2018-11-29 21:01:17,997 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Removing handler for request 100-0000331935 2018-11-29 21:01:17,997 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Removing handler for request 100-0000331936 2018-11-29 21:01:17,998 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Removing handler for request 100-0000331937 2018-11-29 21:01:17,998 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:01:18,192 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:01:18,192 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Declining node request 100-0000331934 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,210 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:01:18,210 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Declining node request 100-0000331935 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,229 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:01:18,229 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Declining node request 100-0000331936 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,252 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:01:18,252 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Declining node request 100-0000331937 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,272 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 100-0000331934 2018-11-29 21:01:18,272 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 100-0000331935 2018-11-29 21:01:18,272 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 100-0000331936 2018-11-29 21:01:18,272 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 100-0000331937 2018-11-29 21:01:18,272 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:01:18,380 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:01:18,381 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Declining node request 100-0000331934 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,416 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:01:18,417 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Declining node request 100-0000331935 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,451 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:01:18,452 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Declining node request 100-0000331936 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,457 INFO nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Assigning node request 2018-11-29 21:01:18,458 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4]: Declining node request 100-0000331934 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,500 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:01:18,500 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Declining node request 100-0000331937 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,503 INFO nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Assigning node request 2018-11-29 21:01:18,503 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4]: Declining node request 100-0000331935 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,526 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Assigning node request 2018-11-29 21:01:18,526 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.small]: Declining node request 100-0000331934 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,547 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 100-0000331934 2018-11-29 21:01:18,547 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 100-0000331935 2018-11-29 21:01:18,547 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 100-0000331936 2018-11-29 21:01:18,547 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 100-0000331937 2018-11-29 21:01:18,547 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:01:18,559 INFO nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Assigning node request 2018-11-29 21:01:18,560 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4]: Declining node request 100-0000331936 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,593 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Assigning node request 2018-11-29 21:01:18,596 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.small]: Declining node request 100-0000331935 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,625 INFO nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Assigning node request 2018-11-29 21:01:18,625 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4]: Declining node request 100-0000331937 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,657 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Assigning node request 2018-11-29 21:01:18,657 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.small]: Declining node request 100-0000331936 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,683 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Removing handler for request 100-0000331934 2018-11-29 21:01:18,683 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Removing handler for request 100-0000331935 2018-11-29 21:01:18,683 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Removing handler for request 100-0000331936 2018-11-29 21:01:18,683 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Removing handler for request 100-0000331937 2018-11-29 21:01:18,683 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:01:18,737 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Assigning node request 2018-11-29 21:01:18,737 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.small]: Declining node request 100-0000331937 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,794 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Removing handler for request 100-0000331934 2018-11-29 21:01:18,794 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Removing handler for request 100-0000331935 2018-11-29 21:01:18,794 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Removing handler for request 100-0000331936 2018-11-29 21:01:18,794 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Removing handler for request 100-0000331937 2018-11-29 21:01:18,795 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:01:18,855 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Assigning node request 2018-11-29 21:01:18,856 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1]: Declining node request 100-0000331934 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,882 INFO nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Assigning node request 2018-11-29 21:01:18,882 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-rdo-ca-ymq-1-main]: Declining node request 100-0000331935 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,916 INFO nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Assigning node request 2018-11-29 21:01:18,917 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Declining node request 100-0000331936 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,958 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Assigning node request 2018-11-29 21:01:18,959 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1]: Declining node request 100-0000331937 because node type(s) [static-ansible] not available 2018-11-29 21:01:18,964 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Removing handler for request 100-0000331935 2018-11-29 21:01:18,964 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:01:18,989 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Removing handler for request 100-0000331936 2018-11-29 21:01:18,990 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:01:18,997 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Assigning node request 2018-11-29 21:01:18,998 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.large]: Declining node request 100-0000331934 because node type(s) [static-ansible] not available 2018-11-29 21:01:19,005 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Removing handler for request 100-0000331934 2018-11-29 21:01:19,005 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Removing handler for request 100-0000331937 2018-11-29 21:01:19,006 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:01:19,040 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 21:01:19,040 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 100-0000331935 because node type(s) [static-ansible] not available 2018-11-29 21:01:19,049 INFO nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Assigning node request 2018-11-29 21:01:19,049 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-tripleo-ovb]: Declining node request 100-0000331936 because node type(s) [static-ansible] not available 2018-11-29 21:01:19,074 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Assigning node request 2018-11-29 21:01:19,074 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.large]: Declining node request 100-0000331937 because node type(s) [static-ansible] not available 2018-11-29 21:01:19,082 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Removing handler for request 100-0000331934 2018-11-29 21:01:19,082 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:01:19,094 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 21:01:19,095 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 100-0000331936 because node type(s) [static-ansible] not available 2018-11-29 21:01:19,102 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Removing handler for request 100-0000331936 2018-11-29 21:01:19,102 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:01:19,111 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Removing handler for request 100-0000331937 2018-11-29 21:01:19,111 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:01:19,125 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 21:01:19,125 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 100-0000331937 because node type(s) [static-ansible] not available 2018-11-29 21:01:19,140 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 100-0000331935 2018-11-29 21:01:19,140 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 100-0000331936 2018-11-29 21:01:19,140 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 100-0000331937 2018-11-29 21:01:19,140 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:01:19,218 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:19,446 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331933 2018-11-29 21:01:19,504 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331933 2018-11-29 21:01:19,553 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331933'] 2018-11-29 21:01:19,617 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.39836907386779785s 2018-11-29 21:01:19,619 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318686, state=deleting, external_id=8c8e2d0b-58dd-4cb8-a764-1fabb5592420 2018-11-29 21:01:20,380 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:20,538 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:21,291 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9107615947723389s 2018-11-29 21:01:21,306 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:01:21,357 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8182358741760254s 2018-11-29 21:01:22,553 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2470035552978516s 2018-11-29 21:01:25,414 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:01:25,669 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2551229000091553s 2018-11-29 21:01:25,670 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:01:25,670 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:01:25,807 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13711261749267578s 2018-11-29 21:01:25,811 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:26,487 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6758806705474854s 2018-11-29 21:01:26,504 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:01:26,504 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:01:26,576 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:26,735 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:27,004 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:01:27,067 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:01:27,067 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:01:27,067 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:01:27,077 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318688 for request 300-0000331925 2018-11-29 21:01:27,084 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:01:27,084 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:01:27,086 INFO nodepool.NodeLauncher-0000318688: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318688 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318688 2018-11-29 21:01:27,086 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:01:27,660 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0840039253234863s 2018-11-29 21:01:27,674 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:01:27,686 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9498794078826904s 2018-11-29 21:01:27,747 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:01:28,057 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:01:28,173 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.0867786407470703s 2018-11-29 21:01:28,174 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServers (queue 0) 2018-11-29 21:01:28,345 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:01:28,521 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServers in 0.347430944442749s 2018-11-29 21:01:28,529 DEBUG nodepool.NodeLauncher-0000318688: Waiting for server df7d1f90-61fb-498f-9932-aa8114656a2c for node id: 0000318688 2018-11-29 21:01:28,623 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:01:28,788 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:01:28,862 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:01:29,062 INFO nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Assigning node request 2018-11-29 21:01:29,063 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-rdo-ca-ymq-1-main]: Declining node request 100-0000331934 because node type(s) [static-ansible] not available 2018-11-29 21:01:29,090 INFO nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Assigning node request 2018-11-29 21:01:29,091 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-rdo-ca-ymq-1-main]: Declining node request 100-0000331936 because node type(s) [static-ansible] not available 2018-11-29 21:01:29,126 INFO nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Assigning node request 2018-11-29 21:01:29,126 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-rdo-ca-ymq-1-main]: Declining node request 100-0000331937 because node type(s) [static-ansible] not available 2018-11-29 21:01:29,148 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4738507270812988s 2018-11-29 21:01:29,169 INFO nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Assigning node request 2018-11-29 21:01:29,170 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Declining node request 100-0000331934 because node type(s) [static-ansible] not available 2018-11-29 21:01:29,185 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Removing handler for request 100-0000331934 2018-11-29 21:01:29,185 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Removing handler for request 100-0000331936 2018-11-29 21:01:29,185 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Removing handler for request 100-0000331937 2018-11-29 21:01:29,185 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:01:29,218 INFO nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Assigning node request 2018-11-29 21:01:29,219 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Declining node request 100-0000331935 because node type(s) [static-ansible] not available 2018-11-29 21:01:29,240 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Assigning node request 2018-11-29 21:01:29,240 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1]: Declining node request 100-0000331936 because node type(s) [static-ansible] not available 2018-11-29 21:01:29,268 INFO nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Assigning node request 2018-11-29 21:01:29,269 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Declining node request 100-0000331937 because node type(s) [static-ansible] not available 2018-11-29 21:01:29,283 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Removing handler for request 100-0000331936 2018-11-29 21:01:29,283 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:01:29,308 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Removing handler for request 100-0000331934 2018-11-29 21:01:29,308 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Removing handler for request 100-0000331935 2018-11-29 21:01:29,308 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Removing handler for request 100-0000331937 2018-11-29 21:01:29,308 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:01:29,352 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Assigning node request 2018-11-29 21:01:29,352 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.large]: Declining node request 100-0000331935 because node type(s) [static-ansible] not available 2018-11-29 21:01:29,383 INFO nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Assigning node request 2018-11-29 21:01:29,385 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-tripleo-ovb]: Declining node request 100-0000331934 because node type(s) [static-ansible] not available 2018-11-29 21:01:29,390 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Assigning node request 2018-11-29 21:01:29,390 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.large]: Declining node request 100-0000331936 because node type(s) [static-ansible] not available 2018-11-29 21:01:29,398 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Assigning node request 2018-11-29 21:01:29,398 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.large]: Declining node request 100-0000331935 because node type(s) [static-ansible] not available 2018-11-29 21:01:29,430 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:01:29,434 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Assigning node request 2018-11-29 21:01:29,435 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.large]: Declining node request 100-0000331937 because node type(s) [static-ansible] not available 2018-11-29 21:01:29,443 INFO nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Assigning node request 2018-11-29 21:01:29,443 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-tripleo-ovb]: Declining node request 100-0000331936 because node type(s) [static-ansible] not available 2018-11-29 21:01:29,451 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Removing handler for request 100-0000331935 2018-11-29 21:01:29,451 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:01:29,464 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Removing handler for request 100-0000331935 2018-11-29 21:01:29,464 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Removing handler for request 100-0000331936 2018-11-29 21:01:29,464 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Removing handler for request 100-0000331937 2018-11-29 21:01:29,464 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:01:29,471 INFO nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Assigning node request 2018-11-29 21:01:29,471 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-tripleo-ovb]: Declining node request 100-0000331937 because node type(s) [static-ansible] not available 2018-11-29 21:01:29,488 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Removing handler for request 100-0000331934 2018-11-29 21:01:29,489 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Removing handler for request 100-0000331936 2018-11-29 21:01:29,489 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Removing handler for request 100-0000331937 2018-11-29 21:01:29,489 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:01:29,555 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331933 2018-11-29 21:01:29,627 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331933 2018-11-29 21:01:29,709 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331933'] 2018-11-29 21:01:31,943 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318687 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:01:31,943 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:01:31,947 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318687, state=deleting, external_id=None 2018-11-29 21:01:32,778 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:33,374 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:33,532 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:33,761 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.982391357421875s 2018-11-29 21:01:34,027 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6529641151428223s 2018-11-29 21:01:34,142 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6100656986236572s 2018-11-29 21:01:34,146 ERROR nodepool.NodeLauncher-0000318688: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318688: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 157, in _launchNode auto_ip=self.pool.auto_floating_ip) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 356, in waitForServer reuse=False, timeout=timeout) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6881, in wait_for_server nat_destination=nat_destination) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6898, in get_active_server "Error in creating the server", extra_data=dict(server=server)) openstack.exceptions.SDKException: Error in creating the server 2018-11-29 21:01:34,155 INFO nodepool.NodeLauncher-0000318688: Request 300-0000331925: Node df7d1f90-61fb-498f-9932-aa8114656a2c scheduled for cleanup 2018-11-29 21:01:34,166 INFO nodepool.NodeLauncher-0000318688: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318688 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318688 2018-11-29 21:01:34,166 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:01:35,175 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:01:35,234 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.068227767944336s 2018-11-29 21:01:35,235 ERROR nodepool.NodeLauncher-0000318688: Request 300-0000331925: Launch attempt 2/3 failed for node 0000318688: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:01:35,235 INFO nodepool.NodeLauncher-0000318688: Quota exceeded, invalidating quota cache 2018-11-29 21:01:35,235 INFO nodepool.NodeLauncher-0000318688: Aborting node 0000318688 due to quota failure 2018-11-29 21:01:36,384 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2094504833221436s 2018-11-29 21:01:37,114 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:01:37,228 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318689 (state: deleting, allocated_to: None) 2018-11-29 21:01:37,228 INFO nodepool.DeletedNodeWorker: Deleting deleting instance df7d1f90-61fb-498f-9932-aa8114656a2c from vexxhost-ansible-network-mtl1 2018-11-29 21:01:37,233 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server df7d1f90-61fb-498f-9932-aa8114656a2c 2018-11-29 21:01:37,338 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.22361993789672852s 2018-11-29 21:01:37,339 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:01:37,339 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeDeleteServers (queue 1) 2018-11-29 21:01:37,837 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:01:38,127 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:01:38,404 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeDeleteServers in 1.0644969940185547s 2018-11-29 21:01:38,404 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:01:38,434 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:01:38,609 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.20500397682189941s 2018-11-29 21:01:38,613 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:01:38,613 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:01:38,692 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:01:38,713 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:01:38,759 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:01:38,858 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:01:38,858 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:01:38,858 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:01:38,872 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:38,880 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318690 for request 300-0000331819 2018-11-29 21:01:38,996 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:01:38,996 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:01:39,000 INFO nodepool.NodeLauncher-0000318690: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318690 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318690 2018-11-29 21:01:39,001 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:01:39,053 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:01:39,097 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:01:39,264 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:01:39,369 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Assigning node request 2018-11-29 21:01:39,369 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1]: Declining node request 100-0000331935 because node type(s) [static-ansible] not available 2018-11-29 21:01:39,393 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Removing handler for request 100-0000331935 2018-11-29 21:01:39,393 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:01:39,412 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:01:39,567 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 21:01:39,570 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 100-0000331934 because node type(s) [static-ansible] not available 2018-11-29 21:01:39,602 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 100-0000331934 2018-11-29 21:01:39,602 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:01:39,619 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Assigning node request 2018-11-29 21:01:39,622 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.large]: Declining node request 100-0000331934 because node type(s) [static-ansible] not available 2018-11-29 21:01:39,651 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Assigning node request 2018-11-29 21:01:39,652 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.large]: Declining node request 100-0000331936 because node type(s) [static-ansible] not available 2018-11-29 21:01:39,652 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:01:39,670 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Removing handler for request 100-0000331934 2018-11-29 21:01:39,670 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Removing handler for request 100-0000331936 2018-11-29 21:01:39,670 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:01:39,678 INFO nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Assigning node request 2018-11-29 21:01:39,678 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-tripleo-ovb]: Declining node request 100-0000331935 because node type(s) [static-ansible] not available 2018-11-29 21:01:39,696 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Removing handler for request 100-0000331935 2018-11-29 21:01:39,696 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:01:39,710 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:39,713 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331933 2018-11-29 21:01:39,774 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331933 2018-11-29 21:01:39,839 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331933'] 2018-11-29 21:01:39,987 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1141698360443115s 2018-11-29 21:01:40,660 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.6587870121002197s 2018-11-29 21:01:40,661 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 1) 2018-11-29 21:01:40,914 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.203876256942749s 2018-11-29 21:01:41,974 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 1.3131403923034668s 2018-11-29 21:01:41,977 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServers (queue 0) 2018-11-29 21:01:41,976 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318689, state=deleting, external_id=df7d1f90-61fb-498f-9932-aa8114656a2c 2018-11-29 21:01:42,415 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:01:42,820 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServers in 0.8428115844726562s 2018-11-29 21:01:42,823 ERROR nodepool.NodeLauncher-0000318690: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318690: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6831, in create_server resource='server', resource_id=server.id) openstack.cloud.exc.OpenStackCloudCreateException: Error creating server: b58ee558-3dbd-4469-99c8-e7dadfed74dc 2018-11-29 21:01:42,843 INFO nodepool.NodeLauncher-0000318690: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318690 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318690 2018-11-29 21:01:42,844 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:01:42,851 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318688 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:01:42,851 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:01:42,855 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318688, state=deleting, external_id=None 2018-11-29 21:01:43,704 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2894842624664307s 2018-11-29 21:01:44,005 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.16105318069458s 2018-11-29 21:01:44,005 ERROR nodepool.NodeLauncher-0000318690: Request 300-0000331819: Launch attempt 2/3 failed for node 0000318690: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:01:44,006 INFO nodepool.NodeLauncher-0000318690: Quota exceeded, invalidating quota cache 2018-11-29 21:01:44,006 INFO nodepool.NodeLauncher-0000318690: Aborting node 0000318690 due to quota failure 2018-11-29 21:01:45,387 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:46,053 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:46,221 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8337106704711914s 2018-11-29 21:01:46,655 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6011002063751221s 2018-11-29 21:01:47,898 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:01:48,186 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:01:48,502 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:01:48,775 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:01:49,000 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:01:49,151 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:01:49,183 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:01:49,241 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.24060940742492676s 2018-11-29 21:01:49,241 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:01:49,242 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:01:49,332 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:01:49,387 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1447300910949707s 2018-11-29 21:01:49,390 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:49,478 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:01:49,497 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:01:49,687 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:01:49,734 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:01:49,774 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:01:49,790 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:01:49,808 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:01:49,841 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331933 2018-11-29 21:01:49,901 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331933 2018-11-29 21:01:49,962 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331933'] 2018-11-29 21:01:50,011 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6208152770996094s 2018-11-29 21:01:50,035 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:01:50,036 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:01:50,169 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:01:50,238 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:01:50,238 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:01:50,238 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:01:50,252 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318691 for request 300-0000331925 2018-11-29 21:01:50,260 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:01:50,260 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:01:50,262 INFO nodepool.NodeLauncher-0000318691: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318691 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318691 2018-11-29 21:01:50,263 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:01:50,324 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318690 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:01:50,324 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:01:50,331 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318690, state=deleting, external_id=None 2018-11-29 21:01:50,456 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:50,808 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.35213780403137207s 2018-11-29 21:01:50,809 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-99be5fec-d211-43c4-ad38-4ba367c94332) 2018-11-29 21:01:50,961 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:01:51,030 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.29561448097229s 2018-11-29 21:01:51,518 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:51,925 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.6619999408721924s 2018-11-29 21:01:51,926 ERROR nodepool.NodeLauncher-0000318691: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318691: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:01:51,926 INFO nodepool.NodeLauncher-0000318691: Quota exceeded, invalidating quota cache 2018-11-29 21:01:51,926 INFO nodepool.NodeLauncher-0000318691: Aborting node 0000318691 due to quota failure 2018-11-29 21:01:52,578 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.05942702293396s 2018-11-29 21:01:52,669 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:53,445 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7754509449005127s 2018-11-29 21:01:57,058 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:01:57,963 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:01:58,248 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:01:58,426 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3676047325134277s 2018-11-29 21:01:58,437 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:58,581 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:01:58,845 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:01:58,950 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:01:59,264 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:01:59,286 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:01:59,413 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:01:59,418 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9761407375335693s 2018-11-29 21:01:59,568 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:01:59,587 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:01:59,768 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8179011344909668s 2018-11-29 21:01:59,781 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:01:59,890 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:01:59,905 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:01:59,918 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:01:59,963 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331933 2018-11-29 21:02:00,037 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331933 2018-11-29 21:02:00,102 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331933'] 2018-11-29 21:02:00,267 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:02:00,403 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.13600707054138184s 2018-11-29 21:02:00,404 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:02:00,404 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:02:00,596 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.19137787818908691s 2018-11-29 21:02:00,599 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:01,174 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5753037929534912s 2018-11-29 21:02:01,192 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:02:01,192 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:02:01,300 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:02:01,361 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:02:01,361 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:02:01,361 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:02:01,373 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318692 for request 300-0000331819 2018-11-29 21:02:01,379 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:02:01,379 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:02:01,381 INFO nodepool.NodeLauncher-0000318692: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318692 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318692 2018-11-29 21:02:01,382 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:02:02,426 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.0441911220550537s 2018-11-29 21:02:02,426 ERROR nodepool.NodeLauncher-0000318692: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318692: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:02:02,427 INFO nodepool.NodeLauncher-0000318692: Quota exceeded, invalidating quota cache 2018-11-29 21:02:02,427 INFO nodepool.NodeLauncher-0000318692: Aborting node 0000318692 due to quota failure 2018-11-29 21:02:04,456 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:05,473 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:05,868 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.4118430614471436s 2018-11-29 21:02:05,884 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:02:06,161 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318691 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:02:06,161 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:02:06,165 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318691, state=deleting, external_id=None 2018-11-29 21:02:06,267 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7945635318756104s 2018-11-29 21:02:07,447 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.5626299381256104s 2018-11-29 21:02:08,031 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:02:08,315 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:02:08,653 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:02:08,923 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:02:09,385 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:02:09,399 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:02:09,489 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:02:09,647 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:02:09,666 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:02:09,860 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:02:10,015 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:02:10,032 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:02:10,038 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:02:10,104 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331933 2018-11-29 21:02:10,164 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331933 2018-11-29 21:02:10,223 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331933'] 2018-11-29 21:02:10,934 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:11,385 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:02:11,531 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.146148681640625s 2018-11-29 21:02:11,532 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:02:11,533 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:02:11,671 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13814973831176758s 2018-11-29 21:02:11,674 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:11,792 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:12,025 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0912656784057617s 2018-11-29 21:02:12,294 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6194736957550049s 2018-11-29 21:02:12,315 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:02:12,315 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:02:12,316 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.5241875648498535s 2018-11-29 21:02:12,433 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:02:12,480 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:02:12,480 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:02:12,480 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:02:12,491 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318693 for request 300-0000331925 2018-11-29 21:02:12,498 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:02:12,499 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:02:12,500 INFO nodepool.NodeLauncher-0000318693: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318693 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318693 2018-11-29 21:02:12,501 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:02:13,475 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:02:13,733 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.231621503829956s 2018-11-29 21:02:13,733 ERROR nodepool.NodeLauncher-0000318693: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318693: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:02:13,736 INFO nodepool.NodeLauncher-0000318693: Quota exceeded, invalidating quota cache 2018-11-29 21:02:13,736 INFO nodepool.NodeLauncher-0000318693: Aborting node 0000318693 due to quota failure 2018-11-29 21:02:14,670 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1945409774780273s 2018-11-29 21:02:16,713 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318692 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:02:16,714 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:02:16,719 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318692, state=deleting, external_id=None 2018-11-29 21:02:17,463 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:18,105 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:02:18,292 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:18,384 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:02:18,718 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2544219493865967s 2018-11-29 21:02:18,740 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:02:18,995 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:02:19,270 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9777204990386963s 2018-11-29 21:02:19,497 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:02:19,507 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:02:19,557 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:02:19,732 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:02:19,753 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:02:19,932 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:02:20,147 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:02:20,161 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:02:20,163 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:02:20,225 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331933 2018-11-29 21:02:20,285 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331933 2018-11-29 21:02:20,360 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331933'] 2018-11-29 21:02:20,699 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:02:21,915 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2155513763427734s 2018-11-29 21:02:22,004 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318664 (state: used, allocated_to: 200-0000331926) 2018-11-29 21:02:22,004 INFO nodepool.DeletedNodeWorker: Deleting used instance None from ansible-statics 2018-11-29 21:02:22,008 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318664, state=deleting, external_id=None 2018-11-29 21:02:23,206 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:02:23,668 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.4620382785797119s 2018-11-29 21:02:23,669 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:02:23,669 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:02:23,695 DEBUG nodepool.driver.static.StaticNodeProvider: Registered static node 35.199.20.133 2018-11-29 21:02:23,753 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:23,799 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1300051212310791s 2018-11-29 21:02:23,803 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:24,341 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:24,515 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.7622921466827393s 2018-11-29 21:02:24,750 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.9475359916687012s 2018-11-29 21:02:24,768 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:02:24,768 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:02:24,860 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:02:24,905 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:02:24,905 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:02:24,905 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:02:24,918 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318695 for request 300-0000331819 2018-11-29 21:02:24,929 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:02:24,929 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:02:24,930 INFO nodepool.NodeLauncher-0000318695: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318695 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318695 2018-11-29 21:02:24,931 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:02:25,035 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6934828758239746s 2018-11-29 21:02:26,009 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.0778510570526123s 2018-11-29 21:02:26,009 ERROR nodepool.NodeLauncher-0000318695: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318695: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:02:26,010 INFO nodepool.NodeLauncher-0000318695: Quota exceeded, invalidating quota cache 2018-11-29 21:02:26,010 INFO nodepool.NodeLauncher-0000318695: Aborting node 0000318695 due to quota failure 2018-11-29 21:02:27,944 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:02:28,183 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:02:28,447 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:02:28,905 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318693 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:02:28,905 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:02:28,911 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318693, state=deleting, external_id=None 2018-11-29 21:02:29,125 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:02:29,202 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:02:29,223 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2783432006835938s 2018-11-29 21:02:29,637 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:02:29,648 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:02:29,679 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:02:29,800 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:29,820 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:02:29,841 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:02:30,027 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:02:30,257 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:02:30,268 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:02:30,270 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:02:30,362 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331933 2018-11-29 21:02:30,490 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unpaused request 2018-11-29 21:02:30,491 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Locked existing node 0000318694 for request 100-0000331933 2018-11-29 21:02:30,498 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Fulfilled node request 100-0000331933 2018-11-29 21:02:30,507 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unlocked node 0000318694 for request 100-0000331933 2018-11-29 21:02:30,526 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 100-0000331933 2018-11-29 21:02:30,526 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:02:30,737 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9372701644897461s 2018-11-29 21:02:31,051 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:31,510 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.4587857723236084s 2018-11-29 21:02:34,933 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:02:35,131 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.19781017303466797s 2018-11-29 21:02:35,132 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:02:35,132 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:02:35,252 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:02:35,331 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.19842863082885742s 2018-11-29 21:02:35,334 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:35,679 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.34485578536987305s 2018-11-29 21:02:35,697 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:02:35,697 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:02:35,804 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:02:35,858 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:02:35,858 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:02:35,858 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:02:35,869 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318696 for request 300-0000331925 2018-11-29 21:02:35,876 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:02:35,877 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:02:35,878 INFO nodepool.NodeLauncher-0000318696: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318696 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318696 2018-11-29 21:02:35,879 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:02:36,565 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3126280307769775s 2018-11-29 21:02:36,581 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:37,019 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.1398098468780518s 2018-11-29 21:02:37,019 ERROR nodepool.NodeLauncher-0000318696: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318696: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:02:37,020 INFO nodepool.NodeLauncher-0000318696: Quota exceeded, invalidating quota cache 2018-11-29 21:02:37,020 INFO nodepool.NodeLauncher-0000318696: Aborting node 0000318696 due to quota failure 2018-11-29 21:02:37,301 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:37,854 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2701640129089355s 2018-11-29 21:02:38,039 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7380080223083496s 2018-11-29 21:02:38,385 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:02:38,508 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:02:39,206 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:02:39,266 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:02:39,504 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318695 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:02:39,505 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:02:39,509 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318695, state=deleting, external_id=None 2018-11-29 21:02:39,923 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:02:39,934 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:02:39,943 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:02:39,990 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:02:39,993 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:02:40,097 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:02:40,390 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:02:40,398 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:02:40,401 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:02:40,599 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:02:40,599 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Accepting node request 100-0000331934 2018-11-29 21:02:40,648 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331934 2018-11-29 21:02:40,648 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Pausing request handling to satisfy request 100-0000331934 2018-11-29 21:02:40,696 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331934'] 2018-11-29 21:02:42,592 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:02:43,533 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:43,803 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2109568119049072s 2018-11-29 21:02:43,819 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:44,499 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9653751850128174s 2018-11-29 21:02:44,951 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1304857730865479s 2018-11-29 21:02:45,884 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:02:46,097 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.21256756782531738s 2018-11-29 21:02:46,097 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:02:46,097 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:02:46,232 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13427066802978516s 2018-11-29 21:02:46,235 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:46,610 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.3746767044067383s 2018-11-29 21:02:46,628 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:02:46,628 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:02:46,731 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:02:46,792 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:02:46,792 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:02:46,793 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:02:46,804 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318697 for request 300-0000331819 2018-11-29 21:02:46,811 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:02:46,811 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:02:46,813 INFO nodepool.NodeLauncher-0000318697: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318697 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318697 2018-11-29 21:02:46,814 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:02:48,067 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.252699613571167s 2018-11-29 21:02:48,067 ERROR nodepool.NodeLauncher-0000318697: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318697: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:02:48,068 INFO nodepool.NodeLauncher-0000318697: Quota exceeded, invalidating quota cache 2018-11-29 21:02:48,068 INFO nodepool.NodeLauncher-0000318697: Aborting node 0000318697 due to quota failure 2018-11-29 21:02:48,619 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:02:48,623 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:02:49,288 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:02:49,350 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:02:49,833 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:02:50,060 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:50,163 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:02:50,179 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:02:50,193 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:02:50,224 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:02:50,227 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:02:50,251 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:02:50,515 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:02:50,522 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:02:50,525 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:02:50,698 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331934 2018-11-29 21:02:50,752 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331934 2018-11-29 21:02:50,774 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318696 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:02:50,774 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:02:50,777 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318696, state=deleting, external_id=None 2018-11-29 21:02:50,927 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331934'] 2018-11-29 21:02:51,383 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:51,576 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.742750644683838s 2018-11-29 21:02:51,592 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:51,646 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.5853819847106934s 2018-11-29 21:02:51,730 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.346813440322876s 2018-11-29 21:02:51,730 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-17452548-846c-4fdc-9e98-605077ec35a7) 2018-11-29 21:02:51,872 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:02:52,575 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9811708927154541s 2018-11-29 21:02:56,816 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:02:57,014 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.19820594787597656s 2018-11-29 21:02:57,015 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:02:57,015 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:02:57,147 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1323528289794922s 2018-11-29 21:02:57,151 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:57,604 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:57,660 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:02:57,898 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.7472047805786133s 2018-11-29 21:02:57,918 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:02:57,918 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:02:58,020 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:02:58,076 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:02:58,077 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:02:58,077 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:02:58,086 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318698 for request 300-0000331925 2018-11-29 21:02:58,093 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:02:58,093 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:02:58,095 INFO nodepool.NodeLauncher-0000318698: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318698 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318698 2018-11-29 21:02:58,095 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:02:58,271 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6112198829650879s 2018-11-29 21:02:58,730 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.126218557357788s 2018-11-29 21:02:58,744 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:02:58,882 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:02:58,883 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:02:59,043 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.947258710861206s 2018-11-29 21:02:59,043 ERROR nodepool.NodeLauncher-0000318698: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318698: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:02:59,043 INFO nodepool.NodeLauncher-0000318698: Quota exceeded, invalidating quota cache 2018-11-29 21:02:59,044 INFO nodepool.NodeLauncher-0000318698: Aborting node 0000318698 due to quota failure 2018-11-29 21:02:59,351 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:02:59,423 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:03:00,230 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.485741376876831s 2018-11-29 21:03:00,440 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:03:00,479 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:03:00,487 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:03:00,512 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:03:00,512 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:03:00,513 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:03:00,649 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:03:00,655 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:03:00,658 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:03:00,929 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331934 2018-11-29 21:03:01,000 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331934 2018-11-29 21:03:01,071 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331934'] 2018-11-29 21:03:01,331 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318697 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:03:01,331 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:03:01,334 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318697, state=deleting, external_id=None 2018-11-29 21:03:03,912 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:04,284 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:04,960 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0479917526245117s 2018-11-29 21:03:04,961 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6766729354858398s 2018-11-29 21:03:06,260 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:03:07,815 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.5550904273986816s 2018-11-29 21:03:08,098 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:03:08,254 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.15596461296081543s 2018-11-29 21:03:08,254 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:03:08,255 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:03:08,460 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.204606294631958s 2018-11-29 21:03:08,463 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:08,807 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.34440088272094727s 2018-11-29 21:03:08,829 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:03:08,830 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:03:09,159 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:03:09,161 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:03:09,171 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:03:09,227 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:03:09,227 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:03:09,227 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:03:09,237 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318699 for request 300-0000331819 2018-11-29 21:03:09,244 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:03:09,244 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:03:09,246 INFO nodepool.NodeLauncher-0000318699: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318699 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318699 2018-11-29 21:03:09,246 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:03:09,423 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:03:09,491 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:03:10,157 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:10,542 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:10,648 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4022314548492432s 2018-11-29 21:03:10,649 ERROR nodepool.NodeLauncher-0000318699: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318699: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:03:10,649 INFO nodepool.NodeLauncher-0000318699: Quota exceeded, invalidating quota cache 2018-11-29 21:03:10,649 INFO nodepool.NodeLauncher-0000318699: Aborting node 0000318699 due to quota failure 2018-11-29 21:03:10,685 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:03:10,807 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:03:10,833 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:03:10,871 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:03:10,876 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:03:10,876 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:03:10,935 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:03:10,935 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:03:10,936 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:03:11,072 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331934 2018-11-29 21:03:11,138 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331934 2018-11-29 21:03:11,208 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331934'] 2018-11-29 21:03:11,444 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2867586612701416s 2018-11-29 21:03:11,652 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.1095490455627441s 2018-11-29 21:03:11,865 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318698 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:03:11,865 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:03:11,868 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318698, state=deleting, external_id=None 2018-11-29 21:03:12,127 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318109 (state: used, allocated_to: 100-0000331932) 2018-11-29 21:03:12,128 INFO nodepool.DeletedNodeWorker: Deleting used instance None from ansible-statics 2018-11-29 21:03:12,130 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318109, state=deleting, external_id=None 2018-11-29 21:03:12,493 DEBUG nodepool.driver.static.StaticNodeProvider: Registered static node 35.236.232.200 2018-11-29 21:03:13,930 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:03:15,461 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.530735731124878s 2018-11-29 21:03:16,472 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:16,994 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:17,401 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.4066786766052246s 2018-11-29 21:03:17,659 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1861412525177002s 2018-11-29 21:03:19,299 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:03:19,421 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:03:19,421 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:03:19,492 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.19288277626037598s 2018-11-29 21:03:19,493 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:03:19,494 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:03:19,522 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:03:19,575 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:03:19,625 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13098430633544922s 2018-11-29 21:03:19,628 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:20,221 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5926041603088379s 2018-11-29 21:03:20,239 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:03:20,240 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:03:20,367 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:03:20,432 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:03:20,432 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:03:20,433 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:03:20,443 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318701 for request 300-0000331925 2018-11-29 21:03:20,451 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:03:20,451 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:03:20,453 INFO nodepool.NodeLauncher-0000318701: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318701 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318701 2018-11-29 21:03:20,454 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:03:20,797 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:03:20,969 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:03:21,068 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:03:21,163 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:03:21,165 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:03:21,168 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:03:21,201 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:03:21,202 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:03:21,203 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:03:21,210 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331934 2018-11-29 21:03:21,369 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unpaused request 2018-11-29 21:03:21,369 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Locked existing node 0000318700 for request 100-0000331934 2018-11-29 21:03:21,375 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Fulfilled node request 100-0000331934 2018-11-29 21:03:21,378 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unlocked node 0000318700 for request 100-0000331934 2018-11-29 21:03:21,388 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 100-0000331934 2018-11-29 21:03:21,388 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:03:21,493 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:03:21,744 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2899048328399658s 2018-11-29 21:03:21,744 ERROR nodepool.NodeLauncher-0000318701: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318701: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:03:21,745 INFO nodepool.NodeLauncher-0000318701: Quota exceeded, invalidating quota cache 2018-11-29 21:03:21,745 INFO nodepool.NodeLauncher-0000318701: Aborting node 0000318701 due to quota failure 2018-11-29 21:03:22,389 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318699 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:03:22,389 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:03:22,392 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318699, state=deleting, external_id=None 2018-11-29 21:03:22,401 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318669 (state: used, allocated_to: 200-0000331927) 2018-11-29 21:03:22,401 INFO nodepool.DeletedNodeWorker: Deleting used instance None from ansible-statics 2018-11-29 21:03:22,407 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318669, state=deleting, external_id=None 2018-11-29 21:03:22,890 DEBUG nodepool.driver.static.StaticNodeProvider: Registered static node 35.221.28.96 2018-11-29 21:03:22,936 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4429714679718018s 2018-11-29 21:03:22,951 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:23,414 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:24,032 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6170766353607178s 2018-11-29 21:03:24,200 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2456920146942139s 2018-11-29 21:03:28,967 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:03:29,688 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:29,738 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:03:29,739 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:03:29,741 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:03:29,757 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:03:30,360 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.392714262008667s 2018-11-29 21:03:30,375 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:30,389 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.701601505279541s 2018-11-29 21:03:30,457 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:03:30,594 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.13685154914855957s 2018-11-29 21:03:30,594 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:03:30,595 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:03:30,800 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.20533180236816406s 2018-11-29 21:03:30,804 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:30,905 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:03:31,036 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:03:31,143 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:03:31,292 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.4882206916809082s 2018-11-29 21:03:31,310 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:03:31,310 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:03:31,405 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:03:31,407 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:03:31,415 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:03:31,470 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0915687084197998s 2018-11-29 21:03:31,489 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:03:31,491 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:03:31,492 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:03:31,544 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:03:31,544 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Accepting node request 100-0000331935 2018-11-29 21:03:31,586 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:03:31,647 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:03:31,647 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:03:31,647 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:03:31,659 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318703 for request 300-0000331819 2018-11-29 21:03:31,665 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:03:31,665 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:03:31,667 INFO nodepool.NodeLauncher-0000318703: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318703 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318703 2018-11-29 21:03:31,667 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:03:31,686 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Locked existing node 0000318702 for request 100-0000331935 2018-11-29 21:03:31,696 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:03:31,697 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Accepting node request 100-0000331936 2018-11-29 21:03:31,767 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331936 2018-11-29 21:03:31,768 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Pausing request handling to satisfy request 100-0000331936 2018-11-29 21:03:31,821 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Fulfilled node request 100-0000331935 2018-11-29 21:03:31,824 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unlocked node 0000318702 for request 100-0000331935 2018-11-29 21:03:31,829 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 100-0000331935 2018-11-29 21:03:31,830 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331936'] 2018-11-29 21:03:32,873 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2052385807037354s 2018-11-29 21:03:32,873 ERROR nodepool.NodeLauncher-0000318703: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318703: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:03:32,873 INFO nodepool.NodeLauncher-0000318703: Quota exceeded, invalidating quota cache 2018-11-29 21:03:32,874 INFO nodepool.NodeLauncher-0000318703: Aborting node 0000318703 due to quota failure 2018-11-29 21:03:33,164 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318701 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:03:33,164 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:03:33,168 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318701, state=deleting, external_id=None 2018-11-29 21:03:36,055 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:36,391 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:03:37,025 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9695806503295898s 2018-11-29 21:03:37,761 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.36985445022583s 2018-11-29 21:03:37,777 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:39,344 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.56406569480896s 2018-11-29 21:03:40,067 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:03:40,068 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:03:40,069 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:03:40,069 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:03:41,000 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:03:41,119 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:03:41,215 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:03:41,556 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:03:41,566 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:03:41,574 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:03:41,661 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:03:41,662 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:03:41,664 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:03:41,669 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:03:41,831 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331936 2018-11-29 21:03:41,878 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331936 2018-11-29 21:03:41,885 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2155611515045166s 2018-11-29 21:03:41,886 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:03:41,886 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:03:41,933 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331936'] 2018-11-29 21:03:42,079 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.193192720413208s 2018-11-29 21:03:42,083 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:42,415 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:42,457 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.37432384490966797s 2018-11-29 21:03:42,475 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:03:42,475 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:03:42,578 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:03:42,632 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:03:42,633 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:03:42,633 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:03:42,648 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318704 for request 300-0000331925 2018-11-29 21:03:42,656 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:03:42,656 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:03:42,657 INFO nodepool.NodeLauncher-0000318704: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318704 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318704 2018-11-29 21:03:42,660 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:03:43,255 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.839893102645874s 2018-11-29 21:03:43,736 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318703 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:03:43,736 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:03:43,738 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318703, state=deleting, external_id=None 2018-11-29 21:03:43,809 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:03:43,828 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.167694330215454s 2018-11-29 21:03:43,828 ERROR nodepool.NodeLauncher-0000318704: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318704: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:03:43,830 INFO nodepool.NodeLauncher-0000318704: Quota exceeded, invalidating quota cache 2018-11-29 21:03:43,830 INFO nodepool.NodeLauncher-0000318704: Aborting node 0000318704 due to quota failure 2018-11-29 21:03:45,267 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4579262733459473s 2018-11-29 21:03:45,283 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:46,486 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.201629638671875s 2018-11-29 21:03:49,052 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:50,107 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.054135799407959s 2018-11-29 21:03:50,413 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:03:50,414 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:03:50,414 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:03:50,415 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:03:51,112 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:03:51,182 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:03:51,285 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:03:51,291 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:03:51,695 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:03:51,725 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:03:51,739 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:03:51,818 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:03:51,819 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:03:51,819 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:03:51,934 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331936 2018-11-29 21:03:52,003 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331936 2018-11-29 21:03:52,074 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331936'] 2018-11-29 21:03:52,315 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:52,623 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.3081786632537842s 2018-11-29 21:03:52,623 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-5bbd1e7f-eecf-4441-993e-719d7ca39986) 2018-11-29 21:03:52,660 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:03:52,762 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:03:52,786 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.494821310043335s 2018-11-29 21:03:52,805 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:52,899 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.23861336708068848s 2018-11-29 21:03:52,900 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:03:52,900 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:03:53,079 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.17917394638061523s 2018-11-29 21:03:53,083 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:53,439 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.35649943351745605s 2018-11-29 21:03:53,457 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:03:53,457 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:03:53,567 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:03:53,617 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:03:53,617 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:03:53,617 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:03:53,628 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318705 for request 300-0000331819 2018-11-29 21:03:53,634 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:03:53,635 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:03:53,638 INFO nodepool.NodeLauncher-0000318705: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318705 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318705 2018-11-29 21:03:53,638 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:03:54,089 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2838895320892334s 2018-11-29 21:03:54,243 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318704 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:03:54,244 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:03:54,248 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318704, state=deleting, external_id=None 2018-11-29 21:03:54,724 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.0854575634002686s 2018-11-29 21:03:54,724 ERROR nodepool.NodeLauncher-0000318705: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318705: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:03:54,724 INFO nodepool.NodeLauncher-0000318705: Quota exceeded, invalidating quota cache 2018-11-29 21:03:54,725 INFO nodepool.NodeLauncher-0000318705: Aborting node 0000318705 due to quota failure 2018-11-29 21:03:55,281 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:03:56,267 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9857547283172607s 2018-11-29 21:03:58,817 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:04:00,101 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2839946746826172s 2018-11-29 21:04:00,116 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:00,874 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:04:00,874 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:04:00,875 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:04:00,876 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:04:01,114 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9977765083312988s 2018-11-29 21:04:01,245 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:04:01,286 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:04:01,360 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:04:01,806 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:04:01,881 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:04:01,912 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:04:01,980 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:04:01,983 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:04:01,984 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:04:02,076 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331936 2018-11-29 21:04:02,130 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331936 2018-11-29 21:04:02,137 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:02,191 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331936'] 2018-11-29 21:04:02,782 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6445932388305664s 2018-11-29 21:04:03,640 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:04:03,898 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.25725841522216797s 2018-11-29 21:04:03,899 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:04:03,899 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:04:04,092 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.19296050071716309s 2018-11-29 21:04:04,096 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:04,656 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5602624416351318s 2018-11-29 21:04:04,674 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:04:04,674 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:04:04,899 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:04:04,977 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:04:04,977 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:04:04,977 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:04:04,992 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318706 for request 300-0000331925 2018-11-29 21:04:05,004 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:04:05,004 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:04:05,006 INFO nodepool.NodeLauncher-0000318706: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318706 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318706 2018-11-29 21:04:05,006 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:04:06,036 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.0289781093597412s 2018-11-29 21:04:06,036 ERROR nodepool.NodeLauncher-0000318706: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318706: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:04:06,037 INFO nodepool.NodeLauncher-0000318706: Quota exceeded, invalidating quota cache 2018-11-29 21:04:06,037 INFO nodepool.NodeLauncher-0000318706: Aborting node 0000318706 due to quota failure 2018-11-29 21:04:06,148 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:07,626 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.4771645069122314s 2018-11-29 21:04:07,639 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:04:08,297 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:08,763 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.46565985679626465s 2018-11-29 21:04:08,938 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2982745170593262s 2018-11-29 21:04:10,078 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318705 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:04:10,078 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:04:10,082 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318705, state=deleting, external_id=None 2018-11-29 21:04:11,209 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:04:11,210 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:04:11,211 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:04:11,211 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:04:11,371 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:04:11,409 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:04:11,454 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:04:11,882 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:04:11,966 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:04:12,010 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:04:12,101 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:04:12,103 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:04:12,104 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:04:12,193 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331936 2018-11-29 21:04:12,261 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331936 2018-11-29 21:04:12,326 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331936'] 2018-11-29 21:04:12,710 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:14,068 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.3577563762664795s 2018-11-29 21:04:14,778 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:14,965 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:04:15,009 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:04:15,145 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.13620424270629883s 2018-11-29 21:04:15,146 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:04:15,147 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:04:15,325 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.5470790863037109s 2018-11-29 21:04:15,345 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.19660520553588867s 2018-11-29 21:04:15,348 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:15,803 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.45454859733581543s 2018-11-29 21:04:15,821 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:04:15,821 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:04:15,942 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:04:16,000 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:04:16,000 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:04:16,000 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:04:16,010 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318707 for request 300-0000331819 2018-11-29 21:04:16,018 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:04:16,018 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:04:16,021 INFO nodepool.NodeLauncher-0000318707: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318707 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318707 2018-11-29 21:04:16,022 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:04:16,276 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.309814453125s 2018-11-29 21:04:17,490 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4674036502838135s 2018-11-29 21:04:17,490 ERROR nodepool.NodeLauncher-0000318707: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318707: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:04:17,491 INFO nodepool.NodeLauncher-0000318707: Quota exceeded, invalidating quota cache 2018-11-29 21:04:17,491 INFO nodepool.NodeLauncher-0000318707: Aborting node 0000318707 due to quota failure 2018-11-29 21:04:19,100 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:19,999 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8985321521759033s 2018-11-29 21:04:20,637 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318706 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:04:20,637 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:04:20,640 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318706, state=deleting, external_id=None 2018-11-29 21:04:20,811 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:21,625 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:04:21,626 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:04:21,628 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:04:21,628 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:04:21,662 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:04:21,675 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:04:21,685 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:04:21,791 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9794955253601074s 2018-11-29 21:04:21,956 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:04:22,050 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:04:22,089 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:04:22,234 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:04:22,236 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:04:22,236 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:04:22,302 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:04:22,328 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331936 2018-11-29 21:04:22,394 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331936 2018-11-29 21:04:22,447 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331936'] 2018-11-29 21:04:23,887 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.5846412181854248s 2018-11-29 21:04:25,130 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:26,023 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:04:26,251 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2278428077697754s 2018-11-29 21:04:26,251 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:04:26,252 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:04:26,304 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1734604835510254s 2018-11-29 21:04:26,442 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.19065499305725098s 2018-11-29 21:04:26,446 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:27,063 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6169819831848145s 2018-11-29 21:04:27,080 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:04:27,081 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:04:27,177 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:04:27,228 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:04:27,228 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:04:27,228 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:04:27,238 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318708 for request 300-0000331925 2018-11-29 21:04:27,246 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:04:27,246 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:04:27,247 INFO nodepool.NodeLauncher-0000318708: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318708 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318708 2018-11-29 21:04:27,248 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:04:27,351 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:28,080 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7282845973968506s 2018-11-29 21:04:28,775 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.5267391204833984s 2018-11-29 21:04:28,775 ERROR nodepool.NodeLauncher-0000318708: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318708: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:04:28,775 INFO nodepool.NodeLauncher-0000318708: Quota exceeded, invalidating quota cache 2018-11-29 21:04:28,776 INFO nodepool.NodeLauncher-0000318708: Aborting node 0000318708 due to quota failure 2018-11-29 21:04:29,916 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:04:31,050 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1344079971313477s 2018-11-29 21:04:31,189 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318707 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:04:31,189 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:04:31,192 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318707, state=deleting, external_id=None 2018-11-29 21:04:31,441 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:32,078 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:04:32,079 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:04:32,079 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:04:32,079 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:04:32,080 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:04:32,082 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:04:32,082 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:04:32,158 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:04:32,180 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:04:32,195 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:04:32,353 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:04:32,354 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:04:32,354 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:04:32,449 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331936 2018-11-29 21:04:32,487 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0450730323791504s 2018-11-29 21:04:32,527 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331936 2018-11-29 21:04:32,594 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331936'] 2018-11-29 21:04:33,812 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:34,722 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9093830585479736s 2018-11-29 21:04:37,082 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:04:37,251 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:04:37,991 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.7395789623260498s 2018-11-29 21:04:37,991 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:04:37,992 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:04:38,116 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.12435197830200195s 2018-11-29 21:04:38,119 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:38,418 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.336381435394287s 2018-11-29 21:04:38,436 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:38,715 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5959022045135498s 2018-11-29 21:04:38,733 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:04:38,734 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:04:38,733 ERROR nodepool.NodeDeleter: Exception deleting instance 08395f7e-438a-42c5-b393-747dc95db233 from rdo-cloud: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 08395f7e-438a-42c5-b393-747dc95db233 deletion 2018-11-29 21:04:38,878 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:04:38,936 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:04:38,936 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:04:38,936 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:04:38,951 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318709 for request 300-0000331819 2018-11-29 21:04:38,960 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:04:38,960 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:04:38,962 INFO nodepool.NodeLauncher-0000318709: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318709 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318709 2018-11-29 21:04:38,962 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:04:39,585 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1485538482666016s 2018-11-29 21:04:39,984 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.021862506866455s 2018-11-29 21:04:39,985 ERROR nodepool.NodeLauncher-0000318709: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318709: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:04:39,986 INFO nodepool.NodeLauncher-0000318709: Quota exceeded, invalidating quota cache 2018-11-29 21:04:39,986 INFO nodepool.NodeLauncher-0000318709: Aborting node 0000318709 due to quota failure 2018-11-29 21:04:40,100 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:41,145 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.0443084239959717s 2018-11-29 21:04:41,731 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318708 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:04:41,731 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:04:41,734 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318708, state=deleting, external_id=None 2018-11-29 21:04:41,756 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000242265 (state: deleting, allocated_to: None) 2018-11-29 21:04:41,756 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 08395f7e-438a-42c5-b393-747dc95db233 from rdo-cloud 2018-11-29 21:04:41,762 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 08395f7e-438a-42c5-b393-747dc95db233 2018-11-29 21:04:41,762 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeDeleteServers (queue 0) 2018-11-29 21:04:41,854 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeDeleteServers in 0.09174656867980957s 2018-11-29 21:04:42,602 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331936 2018-11-29 21:04:42,725 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:04:42,728 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:04:42,729 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:04:42,732 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:04:42,733 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:04:42,734 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:04:42,735 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:04:42,735 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:04:42,735 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:04:42,740 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:04:42,786 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:04:42,787 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:04:42,788 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:04:42,809 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331936 2018-11-29 21:04:42,870 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331936'] 2018-11-29 21:04:44,468 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:04:45,805 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3364126682281494s 2018-11-29 21:04:45,821 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:46,812 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9880497455596924s 2018-11-29 21:04:47,159 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:47,694 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.5347838401794434s 2018-11-29 21:04:48,964 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:04:49,172 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.20784711837768555s 2018-11-29 21:04:49,173 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:04:49,173 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:04:49,312 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13908720016479492s 2018-11-29 21:04:49,316 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:49,753 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.43771839141845703s 2018-11-29 21:04:49,773 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:04:49,773 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:04:49,879 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:04:49,930 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:04:49,931 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:04:49,931 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:04:49,987 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318710 for request 300-0000331925 2018-11-29 21:04:49,994 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:04:49,994 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:04:49,995 INFO nodepool.NodeLauncher-0000318710: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318710 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318710 2018-11-29 21:04:49,996 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:04:51,033 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.0376577377319336s 2018-11-29 21:04:51,034 ERROR nodepool.NodeLauncher-0000318710: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318710: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:04:51,034 INFO nodepool.NodeLauncher-0000318710: Quota exceeded, invalidating quota cache 2018-11-29 21:04:51,034 INFO nodepool.NodeLauncher-0000318710: Aborting node 0000318710 due to quota failure 2018-11-29 21:04:51,833 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:52,271 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318709 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:04:52,271 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:04:52,276 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318709, state=deleting, external_id=None 2018-11-29 21:04:52,864 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0315134525299072s 2018-11-29 21:04:52,895 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:04:52,922 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331936 2018-11-29 21:04:53,504 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331936 2018-11-29 21:04:53,557 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:04:53,557 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:04:53,560 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:04:53,561 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:04:53,562 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:04:53,562 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:04:53,562 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:04:53,563 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:04:53,564 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:04:53,564 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:04:53,564 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:04:53,565 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:04:53,566 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:04:53,623 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331936'] 2018-11-29 21:04:53,711 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:53,855 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:54,181 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.3257114887237549s 2018-11-29 21:04:54,181 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f181c69f-2e2c-4ecd-849d-ac5a6d6c3473) 2018-11-29 21:04:54,306 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:04:54,318 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4235451221466064s 2018-11-29 21:04:54,412 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7014725208282471s 2018-11-29 21:04:57,337 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318644 (state: used, allocated_to: 200-0000331929) 2018-11-29 21:04:57,337 INFO nodepool.DeletedNodeWorker: Deleting used instance None from ansible-statics 2018-11-29 21:04:57,341 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318644, state=deleting, external_id=None 2018-11-29 21:04:57,803 DEBUG nodepool.driver.static.StaticNodeProvider: Registered static node 35.230.187.160 2018-11-29 21:04:58,089 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:04:59,126 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0362818241119385s 2018-11-29 21:04:59,878 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:00,002 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:05:00,233 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.23076963424682617s 2018-11-29 21:05:00,234 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:05:00,234 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:05:00,349 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:05:00,368 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13402771949768066s 2018-11-29 21:05:00,376 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:00,705 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8261394500732422s 2018-11-29 21:05:01,021 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.645026683807373s 2018-11-29 21:05:01,038 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:05:01,038 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:05:01,127 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:05:01,176 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:05:01,177 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:05:01,177 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:05:01,187 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318712 for request 300-0000331819 2018-11-29 21:05:01,196 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:05:01,196 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:05:01,197 INFO nodepool.NodeLauncher-0000318712: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318712 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318712 2018-11-29 21:05:01,198 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:05:01,631 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.281425952911377s 2018-11-29 21:05:02,598 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4003078937530518s 2018-11-29 21:05:02,599 ERROR nodepool.NodeLauncher-0000318712: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318712: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:05:02,599 INFO nodepool.NodeLauncher-0000318712: Quota exceeded, invalidating quota cache 2018-11-29 21:05:02,599 INFO nodepool.NodeLauncher-0000318712: Aborting node 0000318712 due to quota failure 2018-11-29 21:05:02,803 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318710 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:05:02,803 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:05:02,807 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318710, state=deleting, external_id=None 2018-11-29 21:05:03,646 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331936 2018-11-29 21:05:04,169 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:04,251 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:05:04,251 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:05:04,252 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:05:04,253 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:05:04,253 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:05:04,254 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:05:04,254 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:05:04,255 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:05:04,256 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:05:04,256 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:05:04,257 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:05:04,257 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:05:04,258 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:05:04,280 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unpaused request 2018-11-29 21:05:04,281 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Locked existing node 0000318711 for request 100-0000331936 2018-11-29 21:05:04,283 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Fulfilled node request 100-0000331936 2018-11-29 21:05:04,286 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unlocked node 0000318711 for request 100-0000331936 2018-11-29 21:05:04,293 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 100-0000331936 2018-11-29 21:05:04,293 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:05:05,068 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8990485668182373s 2018-11-29 21:05:06,434 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:06,871 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.43746423721313477s 2018-11-29 21:05:07,661 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:05:08,969 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3082423210144043s 2018-11-29 21:05:10,236 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:11,201 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:05:11,218 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9816808700561523s 2018-11-29 21:05:11,349 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.14787793159484863s 2018-11-29 21:05:11,350 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:05:11,350 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:05:11,555 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.20472359657287598s 2018-11-29 21:05:11,559 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:12,174 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6149699687957764s 2018-11-29 21:05:12,192 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:05:12,192 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:05:12,305 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:05:12,358 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:05:12,358 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:05:12,358 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:05:12,369 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318713 for request 300-0000331925 2018-11-29 21:05:12,378 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:05:12,378 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:05:12,380 INFO nodepool.NodeLauncher-0000318713: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318713 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318713 2018-11-29 21:05:12,381 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:05:12,729 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:13,361 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318712 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:05:13,361 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:05:13,364 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318712, state=deleting, external_id=None 2018-11-29 21:05:13,556 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.826873779296875s 2018-11-29 21:05:13,762 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.3812687397003174s 2018-11-29 21:05:13,763 ERROR nodepool.NodeLauncher-0000318713: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318713: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:05:13,763 INFO nodepool.NodeLauncher-0000318713: Quota exceeded, invalidating quota cache 2018-11-29 21:05:13,763 INFO nodepool.NodeLauncher-0000318713: Aborting node 0000318713 due to quota failure 2018-11-29 21:05:14,963 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:05:14,967 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:05:14,968 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:05:14,968 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:05:14,969 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:05:14,970 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:05:14,971 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:05:14,971 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:05:14,972 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:05:14,973 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:05:14,973 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:05:14,975 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:05:14,976 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:05:14,976 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:05:14,976 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Accepting node request 100-0000331937 2018-11-29 21:05:14,998 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:05:15,053 INFO nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Not enough quota remaining to satisfy request 100-0000331937 2018-11-29 21:05:15,053 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Pausing request handling to satisfy request 100-0000331937 2018-11-29 21:05:15,117 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: ['100-0000331937'] 2018-11-29 21:05:16,390 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3911333084106445s 2018-11-29 21:05:16,409 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:17,415 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0060536861419678s 2018-11-29 21:05:18,896 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:19,625 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7282688617706299s 2018-11-29 21:05:22,384 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:05:22,418 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:05:22,529 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.1452648639678955s 2018-11-29 21:05:22,530 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:05:22,530 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:05:22,703 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1725010871887207s 2018-11-29 21:05:22,706 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:23,088 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.3822774887084961s 2018-11-29 21:05:23,107 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:05:23,107 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:05:23,224 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:05:23,284 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:05:23,284 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:05:23,284 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:05:23,334 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318714 for request 300-0000331819 2018-11-29 21:05:23,343 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:05:23,343 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:05:23,345 INFO nodepool.NodeLauncher-0000318714: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318714 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318714 2018-11-29 21:05:23,345 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:05:23,592 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1741461753845215s 2018-11-29 21:05:23,609 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:23,702 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318694 (state: used, allocated_to: 100-0000331933) 2018-11-29 21:05:23,702 INFO nodepool.DeletedNodeWorker: Deleting used instance None from ansible-statics 2018-11-29 21:05:23,706 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318694, state=deleting, external_id=None 2018-11-29 21:05:23,962 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318713 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:05:23,962 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:05:23,965 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318713, state=deleting, external_id=None 2018-11-29 21:05:24,182 DEBUG nodepool.driver.static.StaticNodeProvider: Registered static node 35.199.20.133 2018-11-29 21:05:24,582 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9694526195526123s 2018-11-29 21:05:25,126 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Retrying node request 100-0000331937 2018-11-29 21:05:25,578 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:25,623 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 2.2770087718963623s 2018-11-29 21:05:25,623 ERROR nodepool.NodeLauncher-0000318714: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318714: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:05:25,625 INFO nodepool.NodeLauncher-0000318714: Quota exceeded, invalidating quota cache 2018-11-29 21:05:25,625 INFO nodepool.NodeLauncher-0000318714: Aborting node 0000318714 due to quota failure 2018-11-29 21:05:25,715 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:05:25,716 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:05:25,717 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:05:25,717 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:05:25,719 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:05:25,720 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:05:25,720 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:05:25,721 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:05:25,722 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:05:25,723 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:05:25,723 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:05:25,724 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:05:25,725 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:05:25,794 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unpaused request 2018-11-29 21:05:25,794 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Locked existing node 0000318715 for request 100-0000331937 2018-11-29 21:05:25,798 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Fulfilled node request 100-0000331937 2018-11-29 21:05:25,800 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unlocked node 0000318715 for request 100-0000331937 2018-11-29 21:05:25,806 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 100-0000331937 2018-11-29 21:05:25,806 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:05:26,331 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7523853778839111s 2018-11-29 21:05:29,622 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:30,727 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1049187183380127s 2018-11-29 21:05:30,741 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:05:31,647 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:31,974 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2334599494934082s 2018-11-29 21:05:33,164 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.51607346534729s 2018-11-29 21:05:33,349 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:05:33,569 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2192988395690918s 2018-11-29 21:05:33,569 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:05:33,570 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:05:33,751 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.18143439292907715s 2018-11-29 21:05:33,755 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:34,098 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.342451810836792s 2018-11-29 21:05:34,117 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:05:34,118 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:05:34,214 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:05:34,265 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:05:34,265 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:05:34,265 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:05:34,275 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318716 for request 300-0000331925 2018-11-29 21:05:34,281 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:05:34,281 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:05:34,283 INFO nodepool.NodeLauncher-0000318716: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318716 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318716 2018-11-29 21:05:34,283 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:05:34,573 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318714 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:05:34,573 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:05:34,576 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318714, state=deleting, external_id=None 2018-11-29 21:05:35,163 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.8793301582336426s 2018-11-29 21:05:35,163 ERROR nodepool.NodeLauncher-0000318716: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318716: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:05:35,164 INFO nodepool.NodeLauncher-0000318716: Quota exceeded, invalidating quota cache 2018-11-29 21:05:35,164 INFO nodepool.NodeLauncher-0000318716: Aborting node 0000318716 due to quota failure 2018-11-29 21:05:36,000 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:36,395 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:05:36,395 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:05:36,397 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:05:36,398 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:05:36,399 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:05:36,399 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:05:36,399 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:05:36,400 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:05:36,400 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:05:36,400 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:05:36,401 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:05:36,402 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:05:36,403 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:05:36,406 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:05:37,564 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.5636165142059326s 2018-11-29 21:05:37,577 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:05:38,356 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:38,879 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3017349243164062s 2018-11-29 21:05:39,015 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6593334674835205s 2018-11-29 21:05:42,623 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:43,936 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.3131942749023438s 2018-11-29 21:05:44,286 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:05:44,479 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.19193792343139648s 2018-11-29 21:05:44,479 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:05:44,479 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:05:44,606 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.12654662132263184s 2018-11-29 21:05:44,609 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:44,909 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:05:44,958 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.3487544059753418s 2018-11-29 21:05:44,977 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:05:44,977 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:05:45,031 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:45,241 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:05:45,314 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:05:45,314 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:05:45,315 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:05:45,326 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318717 for request 300-0000331819 2018-11-29 21:05:45,333 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:05:45,333 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:05:45,335 INFO nodepool.NodeLauncher-0000318717: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318717 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318717 2018-11-29 21:05:45,336 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:05:46,037 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.0053796768188477s 2018-11-29 21:05:46,110 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2011902332305908s 2018-11-29 21:05:46,385 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.0491244792938232s 2018-11-29 21:05:46,386 ERROR nodepool.NodeLauncher-0000318717: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318717: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:05:46,507 INFO nodepool.NodeLauncher-0000318717: Quota exceeded, invalidating quota cache 2018-11-29 21:05:46,507 INFO nodepool.NodeLauncher-0000318717: Aborting node 0000318717 due to quota failure 2018-11-29 21:05:47,263 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:05:47,264 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:05:47,264 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:05:47,266 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:05:47,267 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:05:47,267 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:05:47,267 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:05:47,267 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:05:47,268 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:05:47,268 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:05:47,268 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:05:47,269 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:05:47,269 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:05:47,270 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:05:49,058 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:50,210 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.152026653289795s 2018-11-29 21:05:50,412 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318716 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:05:50,412 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:05:50,415 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318716, state=deleting, external_id=None 2018-11-29 21:05:51,191 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:52,062 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8704850673675537s 2018-11-29 21:05:52,139 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:05:53,387 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2475993633270264s 2018-11-29 21:05:54,714 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:55,135 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.4212675094604492s 2018-11-29 21:05:55,135 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-17da1b94-c1c4-4a4b-8d07-fae4be6011f0) 2018-11-29 21:05:55,273 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:05:55,297 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:55,339 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:05:55,551 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2115342617034912s 2018-11-29 21:05:55,551 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:05:55,552 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:05:55,685 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13297486305236816s 2018-11-29 21:05:55,688 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:56,162 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8649132251739502s 2018-11-29 21:05:56,266 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5773639678955078s 2018-11-29 21:05:56,283 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:05:56,283 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:05:56,387 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:05:56,440 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:05:56,440 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:05:56,440 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:05:56,451 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318718 for request 300-0000331925 2018-11-29 21:05:56,458 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:05:56,459 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:05:56,460 INFO nodepool.NodeLauncher-0000318718: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318718 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318718 2018-11-29 21:05:56,461 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:05:57,902 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4411778450012207s 2018-11-29 21:05:57,903 ERROR nodepool.NodeLauncher-0000318718: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318718: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:05:57,904 INFO nodepool.NodeLauncher-0000318718: Quota exceeded, invalidating quota cache 2018-11-29 21:05:57,904 INFO nodepool.NodeLauncher-0000318718: Aborting node 0000318718 due to quota failure 2018-11-29 21:05:58,017 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:05:58,017 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:05:58,017 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:05:58,019 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:05:58,019 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:05:58,020 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:05:58,021 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:05:58,021 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:05:58,022 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:05:58,022 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:05:58,022 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:05:58,023 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:05:58,023 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:05:58,024 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:05:58,063 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:05:58,826 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7635095119476318s 2018-11-29 21:05:59,417 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:06:00,712 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2944257259368896s 2018-11-29 21:06:00,963 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318717 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:06:00,963 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:06:00,967 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318717, state=deleting, external_id=None 2018-11-29 21:06:01,238 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:02,713 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.475597858428955s 2018-11-29 21:06:04,084 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:05,061 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9768109321594238s 2018-11-29 21:06:06,463 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:06:06,671 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.20788240432739258s 2018-11-29 21:06:06,672 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:06:06,672 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:06:06,741 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:06:06,852 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1801280975341797s 2018-11-29 21:06:06,856 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:07,265 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.4092230796813965s 2018-11-29 21:06:07,284 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:06:07,284 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:06:07,382 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:06:07,430 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:06:07,430 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:06:07,430 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:06:07,441 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318719 for request 300-0000331819 2018-11-29 21:06:07,448 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:06:07,448 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:06:07,449 INFO nodepool.NodeLauncher-0000318719: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318719 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318719 2018-11-29 21:06:07,450 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:06:07,947 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2059121131896973s 2018-11-29 21:06:07,963 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:08,246 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.7959067821502686s 2018-11-29 21:06:08,247 ERROR nodepool.NodeLauncher-0000318719: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318719: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:06:08,248 INFO nodepool.NodeLauncher-0000318719: Quota exceeded, invalidating quota cache 2018-11-29 21:06:08,248 INFO nodepool.NodeLauncher-0000318719: Aborting node 0000318719 due to quota failure 2018-11-29 21:06:08,754 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:06:08,755 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:06:08,756 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:06:08,758 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:06:08,759 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:06:08,759 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:06:08,759 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:06:08,760 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:06:08,760 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:06:08,761 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:06:08,762 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:06:08,762 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:06:08,762 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:06:08,763 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:06:08,976 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.009605884552002s 2018-11-29 21:06:10,852 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:11,502 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318718 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:06:11,503 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:06:11,506 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318718, state=deleting, external_id=None 2018-11-29 21:06:11,558 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7061982154846191s 2018-11-29 21:06:13,979 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:06:15,305 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3260653018951416s 2018-11-29 21:06:15,320 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:16,216 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8925952911376953s 2018-11-29 21:06:17,089 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:17,454 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:06:17,589 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.13502144813537598s 2018-11-29 21:06:17,590 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:06:17,590 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:06:17,729 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13874053955078125s 2018-11-29 21:06:17,733 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:17,882 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.793285608291626s 2018-11-29 21:06:18,468 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.735327959060669s 2018-11-29 21:06:18,486 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:06:18,487 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:06:18,593 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:06:18,645 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:06:18,646 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:06:18,646 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:06:18,656 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318720 for request 300-0000331925 2018-11-29 21:06:18,663 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:06:18,663 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:06:18,664 INFO nodepool.NodeLauncher-0000318720: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318720 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318720 2018-11-29 21:06:18,665 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:06:19,475 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:06:19,476 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:06:19,477 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:06:19,478 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:06:19,479 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:06:19,480 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:06:19,480 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:06:19,481 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:06:19,481 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:06:19,481 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:06:19,482 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:06:19,482 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:06:19,482 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:06:19,483 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:06:19,900 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2350718975067139s 2018-11-29 21:06:19,900 ERROR nodepool.NodeLauncher-0000318720: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318720: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:06:19,901 INFO nodepool.NodeLauncher-0000318720: Quota exceeded, invalidating quota cache 2018-11-29 21:06:19,901 INFO nodepool.NodeLauncher-0000318720: Aborting node 0000318720 due to quota failure 2018-11-29 21:06:21,330 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:22,036 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318719 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:06:22,036 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:06:22,039 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318719, state=deleting, external_id=None 2018-11-29 21:06:22,695 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.3651280403137207s 2018-11-29 21:06:22,708 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:06:22,767 ERROR nodepool.NodeDeleter: Exception deleting instance b8f745b0-ef71-49e0-b432-3868667292b9 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server b8f745b0-ef71-49e0-b432-3868667292b9 deletion 2018-11-29 21:06:23,580 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:24,078 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3699703216552734s 2018-11-29 21:06:24,228 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6479387283325195s 2018-11-29 21:06:27,149 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211292 (state: deleting, allocated_to: None) 2018-11-29 21:06:27,149 INFO nodepool.DeletedNodeWorker: Deleting deleting instance b8f745b0-ef71-49e0-b432-3868667292b9 from rdo-cloud-tripleo 2018-11-29 21:06:27,154 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server b8f745b0-ef71-49e0-b432-3868667292b9 2018-11-29 21:06:27,155 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:06:27,227 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.07193660736083984s 2018-11-29 21:06:28,101 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:28,667 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:06:28,810 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.14231419563293457s 2018-11-29 21:06:28,810 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:06:28,810 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:06:29,017 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.20645761489868164s 2018-11-29 21:06:29,021 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:29,169 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.067770004272461s 2018-11-29 21:06:29,192 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:06:29,679 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6585605144500732s 2018-11-29 21:06:29,698 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:06:29,699 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:06:29,908 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:30,285 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:06:30,292 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:06:30,294 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:06:30,295 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:06:30,296 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:06:30,297 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:06:30,298 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:06:30,300 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:06:30,301 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:06:30,302 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:06:30,302 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:06:30,303 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:06:30,307 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:06:30,307 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:06:30,369 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:06:30,429 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:06:30,429 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:06:30,429 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:06:30,441 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318721 for request 300-0000331819 2018-11-29 21:06:30,447 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:06:30,447 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:06:30,448 INFO nodepool.NodeLauncher-0000318721: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318721 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318721 2018-11-29 21:06:30,449 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:06:30,600 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4073843955993652s 2018-11-29 21:06:30,797 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8885834217071533s 2018-11-29 21:06:31,244 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.7950382232666016s 2018-11-29 21:06:31,244 ERROR nodepool.NodeLauncher-0000318721: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318721: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:06:31,245 INFO nodepool.NodeLauncher-0000318721: Quota exceeded, invalidating quota cache 2018-11-29 21:06:31,245 INFO nodepool.NodeLauncher-0000318721: Aborting node 0000318721 due to quota failure 2018-11-29 21:06:32,377 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318670 (state: used, allocated_to: 200-0000331928) 2018-11-29 21:06:32,377 INFO nodepool.DeletedNodeWorker: Deleting used instance None from ansible-statics 2018-11-29 21:06:32,380 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318670, state=deleting, external_id=None 2018-11-29 21:06:32,587 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318720 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:06:32,588 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:06:32,590 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318720, state=deleting, external_id=None 2018-11-29 21:06:32,867 DEBUG nodepool.driver.static.StaticNodeProvider: Registered static node 35.221.20.172 2018-11-29 21:06:34,237 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:35,012 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.775254487991333s 2018-11-29 21:06:36,250 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:36,628 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:06:36,989 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7387688159942627s 2018-11-29 21:06:38,090 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4619123935699463s 2018-11-29 21:06:40,110 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:40,565 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:06:40,769 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.20376205444335938s 2018-11-29 21:06:40,769 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:06:40,770 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:06:40,908 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13840770721435547s 2018-11-29 21:06:40,912 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:41,164 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:06:41,165 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:06:41,165 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:06:41,166 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:06:41,167 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:06:41,168 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:06:41,168 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:06:41,168 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:06:41,169 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:06:41,169 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:06:41,169 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:06:41,170 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:06:41,170 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:06:41,170 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:06:41,199 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0882441997528076s 2018-11-29 21:06:41,562 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6495976448059082s 2018-11-29 21:06:41,580 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:06:41,580 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:06:41,673 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:06:41,720 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:06:41,720 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:06:41,721 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:06:41,730 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318723 for request 300-0000331925 2018-11-29 21:06:41,739 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:06:41,739 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:06:41,740 INFO nodepool.NodeLauncher-0000318723: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318723 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318723 2018-11-29 21:06:41,741 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:06:42,821 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:43,087 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.3461763858795166s 2018-11-29 21:06:43,087 ERROR nodepool.NodeLauncher-0000318723: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318723: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:06:43,088 INFO nodepool.NodeLauncher-0000318723: Quota exceeded, invalidating quota cache 2018-11-29 21:06:43,088 INFO nodepool.NodeLauncher-0000318723: Aborting node 0000318723 due to quota failure 2018-11-29 21:06:43,091 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318721 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:06:43,091 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:06:43,095 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318721, state=deleting, external_id=None 2018-11-29 21:06:43,215 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:06:43,873 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.0524036884307861s 2018-11-29 21:06:44,431 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2152445316314697s 2018-11-29 21:06:46,281 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:47,328 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0468389987945557s 2018-11-29 21:06:48,202 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318639 (state: used, allocated_to: 200-0000331930) 2018-11-29 21:06:48,202 INFO nodepool.DeletedNodeWorker: Deleting used instance None from ansible-statics 2018-11-29 21:06:48,207 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318639, state=deleting, external_id=None 2018-11-29 21:06:48,689 DEBUG nodepool.driver.static.StaticNodeProvider: Registered static node 35.221.27.151 2018-11-29 21:06:49,012 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:49,860 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8484935760498047s 2018-11-29 21:06:50,459 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:06:51,691 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2309160232543945s 2018-11-29 21:06:51,774 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:06:51,879 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:06:51,879 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:06:51,880 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:06:51,881 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:06:51,881 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:06:51,882 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:06:51,882 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:06:51,882 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:06:51,883 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:06:51,883 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:06:51,884 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:06:51,885 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:06:51,886 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:06:51,888 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:06:51,939 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.16434979438781738s 2018-11-29 21:06:51,940 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:06:51,940 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:06:52,077 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13673138618469238s 2018-11-29 21:06:52,081 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:52,474 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.39299654960632324s 2018-11-29 21:06:52,492 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:06:52,492 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:06:52,607 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:06:52,613 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:52,662 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:06:52,662 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:06:52,662 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:06:52,673 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318725 for request 300-0000331819 2018-11-29 21:06:52,679 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:06:52,680 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:06:52,681 INFO nodepool.NodeLauncher-0000318725: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318725 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318725 2018-11-29 21:06:52,682 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:06:53,651 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318723 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:06:53,651 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:06:53,654 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318723, state=deleting, external_id=None 2018-11-29 21:06:53,725 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1111841201782227s 2018-11-29 21:06:54,231 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.5490367412567139s 2018-11-29 21:06:54,231 ERROR nodepool.NodeLauncher-0000318725: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318725: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:06:54,232 INFO nodepool.NodeLauncher-0000318725: Quota exceeded, invalidating quota cache 2018-11-29 21:06:54,232 INFO nodepool.NodeLauncher-0000318725: Aborting node 0000318725 due to quota failure 2018-11-29 21:06:55,713 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:55,875 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:06:56,055 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.3417778015136719s 2018-11-29 21:06:56,055 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-9b0683b9-941b-4898-98dd-0bef1f4eacfb) 2018-11-29 21:06:56,181 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:06:56,976 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.1009809970855713s 2018-11-29 21:06:57,720 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:06:59,170 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4494307041168213s 2018-11-29 21:06:59,185 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:00,299 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1114585399627686s 2018-11-29 21:07:02,625 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:07:02,626 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:07:02,626 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:07:02,628 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:07:02,628 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:07:02,629 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:07:02,629 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:07:02,630 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:07:02,630 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:07:02,630 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:07:02,631 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:07:02,631 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:07:02,631 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:07:02,632 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:07:02,684 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:07:02,814 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.12965130805969238s 2018-11-29 21:07:02,815 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:07:02,815 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:07:02,990 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:03,019 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.2038412094116211s 2018-11-29 21:07:03,022 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:03,629 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6061458587646484s 2018-11-29 21:07:03,647 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:07:03,647 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:07:03,788 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7979464530944824s 2018-11-29 21:07:03,801 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:07:03,857 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:07:03,857 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:07:03,857 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:07:03,868 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318726 for request 300-0000331925 2018-11-29 21:07:03,877 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:07:03,877 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:07:03,879 INFO nodepool.NodeLauncher-0000318726: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318726 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318726 2018-11-29 21:07:03,880 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:07:04,198 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318725 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:07:04,198 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:07:04,202 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318725, state=deleting, external_id=None 2018-11-29 21:07:05,147 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2670478820800781s 2018-11-29 21:07:05,147 ERROR nodepool.NodeLauncher-0000318726: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318726: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:07:05,148 INFO nodepool.NodeLauncher-0000318726: Quota exceeded, invalidating quota cache 2018-11-29 21:07:05,148 INFO nodepool.NodeLauncher-0000318726: Aborting node 0000318726 due to quota failure 2018-11-29 21:07:05,196 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:07:06,505 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3083155155181885s 2018-11-29 21:07:06,520 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:07,525 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0024137496948242s 2018-11-29 21:07:09,282 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318651 (state: used, allocated_to: 200-0000331931) 2018-11-29 21:07:09,282 INFO nodepool.DeletedNodeWorker: Deleting used instance None from ansible-statics 2018-11-29 21:07:09,288 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318651, state=deleting, external_id=None 2018-11-29 21:07:09,756 DEBUG nodepool.driver.static.StaticNodeProvider: Registered static node 35.186.186.222 2018-11-29 21:07:09,799 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:10,628 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8281955718994141s 2018-11-29 21:07:11,767 ERROR nodepool.NodeDeleter: Exception deleting instance 125c7ce6-fd70-4a46-85c8-b6728da772e5 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 125c7ce6-fd70-4a46-85c8-b6728da772e5 deletion 2018-11-29 21:07:12,535 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:07:13,321 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:07:13,322 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:07:13,322 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:07:13,323 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:07:13,324 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:07:13,324 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:07:13,325 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:07:13,325 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:07:13,326 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:07:13,327 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:07:13,327 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:07:13,327 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:07:13,328 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:07:13,328 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:07:13,881 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:07:14,079 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.19753479957580566s 2018-11-29 21:07:14,080 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:07:14,080 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:07:14,141 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.605266809463501s 2018-11-29 21:07:14,156 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:14,237 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.15671801567077637s 2018-11-29 21:07:14,240 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:14,587 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211296 (state: deleting, allocated_to: None) 2018-11-29 21:07:14,587 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 125c7ce6-fd70-4a46-85c8-b6728da772e5 from rdo-cloud-tripleo 2018-11-29 21:07:14,592 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 125c7ce6-fd70-4a46-85c8-b6728da772e5 2018-11-29 21:07:15,045 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.8042294979095459s 2018-11-29 21:07:15,063 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:07:15,064 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:07:15,074 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9159743785858154s 2018-11-29 21:07:15,088 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:07:15,160 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.0717918872833252s 2018-11-29 21:07:15,207 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:07:15,270 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:07:15,270 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:07:15,270 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:07:15,280 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318728 for request 300-0000331819 2018-11-29 21:07:15,289 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:07:15,290 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:07:15,293 INFO nodepool.NodeLauncher-0000318728: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318728 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318728 2018-11-29 21:07:15,294 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:07:15,937 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:16,528 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2346470355987549s 2018-11-29 21:07:16,529 ERROR nodepool.NodeLauncher-0000318728: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318728: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:07:16,530 INFO nodepool.NodeLauncher-0000318728: Quota exceeded, invalidating quota cache 2018-11-29 21:07:16,530 INFO nodepool.NodeLauncher-0000318728: Aborting node 0000318728 due to quota failure 2018-11-29 21:07:16,831 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8942849636077881s 2018-11-29 21:07:20,018 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318726 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:07:20,018 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:07:20,022 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318726, state=deleting, external_id=None 2018-11-29 21:07:20,169 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:21,049 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8799769878387451s 2018-11-29 21:07:21,063 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:07:22,425 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3615269660949707s 2018-11-29 21:07:22,656 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:24,022 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:07:24,023 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:07:24,024 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:07:24,024 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:07:24,024 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:07:24,025 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:07:24,025 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:07:24,025 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:07:24,026 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:07:24,026 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:07:24,027 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:07:24,027 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:07:24,027 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:07:24,028 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:07:24,199 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.5432260036468506s 2018-11-29 21:07:25,303 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:07:25,542 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.23955535888671875s 2018-11-29 21:07:25,543 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:07:25,543 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:07:25,671 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.12789368629455566s 2018-11-29 21:07:25,675 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:26,252 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5773630142211914s 2018-11-29 21:07:26,270 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:07:26,270 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:07:26,357 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:26,390 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:07:26,446 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:07:26,446 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:07:26,447 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:07:26,459 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318729 for request 300-0000331925 2018-11-29 21:07:26,466 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:07:26,467 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:07:26,468 INFO nodepool.NodeLauncher-0000318729: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318729 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318729 2018-11-29 21:07:26,469 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:07:27,298 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.8288297653198242s 2018-11-29 21:07:27,298 ERROR nodepool.NodeLauncher-0000318729: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318729: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:07:27,299 INFO nodepool.NodeLauncher-0000318729: Quota exceeded, invalidating quota cache 2018-11-29 21:07:27,299 INFO nodepool.NodeLauncher-0000318729: Aborting node 0000318729 due to quota failure 2018-11-29 21:07:27,876 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.5189316272735596s 2018-11-29 21:07:28,454 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:07:29,787 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3328280448913574s 2018-11-29 21:07:30,215 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:30,547 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318728 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:07:30,547 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:07:30,551 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318728, state=deleting, external_id=None 2018-11-29 21:07:30,949 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7331275939941406s 2018-11-29 21:07:33,119 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:34,214 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0948512554168701s 2018-11-29 21:07:34,818 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:07:34,821 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:07:34,822 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:07:34,822 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:07:34,824 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:07:34,824 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:07:34,824 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:07:34,825 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:07:34,825 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:07:34,826 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:07:34,826 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:07:34,826 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:07:34,827 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:07:34,827 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:07:35,817 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:07:36,471 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:07:36,667 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.19556522369384766s 2018-11-29 21:07:36,668 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:07:36,668 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:07:36,869 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:36,877 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.20879411697387695s 2018-11-29 21:07:36,881 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:37,332 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.514822244644165s 2018-11-29 21:07:37,564 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6827876567840576s 2018-11-29 21:07:37,581 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:07:37,582 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:07:37,702 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:07:37,753 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8833022117614746s 2018-11-29 21:07:37,761 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:07:37,761 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:07:37,761 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:07:37,769 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318730 for request 300-0000331819 2018-11-29 21:07:37,775 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:07:37,776 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:07:37,777 INFO nodepool.NodeLauncher-0000318730: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318730 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318730 2018-11-29 21:07:37,778 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:07:38,924 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.1453723907470703s 2018-11-29 21:07:38,924 ERROR nodepool.NodeLauncher-0000318730: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318730: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:07:38,926 INFO nodepool.NodeLauncher-0000318730: Quota exceeded, invalidating quota cache 2018-11-29 21:07:38,926 INFO nodepool.NodeLauncher-0000318730: Aborting node 0000318730 due to quota failure 2018-11-29 21:07:39,252 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:40,261 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0083308219909668s 2018-11-29 21:07:41,046 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318729 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:07:41,046 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:07:41,049 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318729, state=deleting, external_id=None 2018-11-29 21:07:42,969 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:43,365 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:07:43,819 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8496642112731934s 2018-11-29 21:07:44,499 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1331748962402344s 2018-11-29 21:07:45,348 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:45,508 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:07:45,509 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:07:45,509 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:07:45,510 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:07:45,511 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:07:45,511 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:07:45,511 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:07:45,512 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:07:45,513 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:07:45,513 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:07:45,514 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:07:45,514 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:07:45,514 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:07:45,515 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:07:46,308 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318702 (state: used, allocated_to: 100-0000331935) 2018-11-29 21:07:46,308 INFO nodepool.DeletedNodeWorker: Deleting used instance None from ansible-statics 2018-11-29 21:07:46,312 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318702, state=deleting, external_id=None 2018-11-29 21:07:46,510 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1615705490112305s 2018-11-29 21:07:46,702 DEBUG nodepool.driver.static.StaticNodeProvider: Registered static node 35.221.28.96 2018-11-29 21:07:47,781 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:07:47,996 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.21455049514770508s 2018-11-29 21:07:47,996 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:07:47,997 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:07:48,194 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1971731185913086s 2018-11-29 21:07:48,197 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:48,586 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.38820314407348633s 2018-11-29 21:07:48,604 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:07:48,604 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:07:48,699 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:07:48,750 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:07:48,750 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:07:48,750 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:07:48,762 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318732 for request 300-0000331925 2018-11-29 21:07:48,769 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:07:48,769 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:07:48,771 INFO nodepool.NodeLauncher-0000318732: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318732 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318732 2018-11-29 21:07:48,772 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:07:49,776 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:50,114 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.3417909145355225s 2018-11-29 21:07:50,114 ERROR nodepool.NodeLauncher-0000318732: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318732: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:07:50,115 INFO nodepool.NodeLauncher-0000318732: Quota exceeded, invalidating quota cache 2018-11-29 21:07:50,115 INFO nodepool.NodeLauncher-0000318732: Aborting node 0000318732 due to quota failure 2018-11-29 21:07:50,334 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.5573184490203857s 2018-11-29 21:07:50,533 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:07:51,537 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318730 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:07:51,537 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:07:51,540 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318730, state=deleting, external_id=None 2018-11-29 21:07:51,715 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1821966171264648s 2018-11-29 21:07:51,731 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:52,452 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.7188737392425537s 2018-11-29 21:07:53,158 ERROR nodepool.NodeDeleter: Exception deleting instance d075965c-a5c0-41e0-97bc-70050e66726e from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server d075965c-a5c0-41e0-97bc-70050e66726e deletion 2018-11-29 21:07:55,843 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:56,218 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:07:56,220 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:07:56,221 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:07:56,222 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:07:56,222 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:07:56,223 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:07:56,223 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:07:56,223 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:07:56,224 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:07:56,224 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:07:56,225 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:07:56,225 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:07:56,225 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:07:56,225 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:07:56,591 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:56,633 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211291 (state: deleting, allocated_to: None) 2018-11-29 21:07:56,633 INFO nodepool.DeletedNodeWorker: Deleting deleting instance d075965c-a5c0-41e0-97bc-70050e66726e from rdo-cloud-tripleo 2018-11-29 21:07:56,640 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server d075965c-a5c0-41e0-97bc-70050e66726e 2018-11-29 21:07:56,641 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:07:57,096 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.5045714378356934s 2018-11-29 21:07:57,096 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-e61bcd27-d0d1-449c-ba85-0906a853ea56) 2018-11-29 21:07:57,251 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:07:57,474 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.832594633102417s 2018-11-29 21:07:57,474 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:58,004 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 2.16060471534729s 2018-11-29 21:07:58,608 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1340465545654297s 2018-11-29 21:07:58,622 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:07:58,774 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:07:58,986 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2117929458618164s 2018-11-29 21:07:58,986 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:07:58,987 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:07:59,143 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.15640854835510254s 2018-11-29 21:07:59,147 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:07:59,466 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.319779634475708s 2018-11-29 21:07:59,484 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:07:59,485 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:07:59,590 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:07:59,639 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:07:59,639 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:07:59,639 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:07:59,651 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318733 for request 300-0000331819 2018-11-29 21:07:59,657 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:07:59,657 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:07:59,659 INFO nodepool.NodeLauncher-0000318733: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318733 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318733 2018-11-29 21:07:59,659 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:07:59,892 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.270606279373169s 2018-11-29 21:08:00,810 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.1505131721496582s 2018-11-29 21:08:00,811 ERROR nodepool.NodeLauncher-0000318733: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318733: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:08:00,811 INFO nodepool.NodeLauncher-0000318733: Quota exceeded, invalidating quota cache 2018-11-29 21:08:00,811 INFO nodepool.NodeLauncher-0000318733: Aborting node 0000318733 due to quota failure 2018-11-29 21:08:02,023 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318732 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:08:02,023 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:08:02,026 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318732, state=deleting, external_id=None 2018-11-29 21:08:03,725 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:04,024 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:04,664 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.639812707901001s 2018-11-29 21:08:04,825 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0995683670043945s 2018-11-29 21:08:05,928 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:08:06,932 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:08:06,935 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:08:06,936 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:08:06,937 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:08:06,937 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:08:06,938 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:08:06,939 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:08:06,939 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:08:06,940 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:08:06,940 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:08:06,940 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:08:06,941 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:08:06,941 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:08:06,941 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:08:07,069 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1410102844238281s 2018-11-29 21:08:08,318 ERROR nodepool.NodeDeleter: Exception deleting instance d0021aac-4e31-42c5-a6a6-b71097532a56 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server d0021aac-4e31-42c5-a6a6-b71097532a56 deletion 2018-11-29 21:08:09,662 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:08:09,886 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2237405776977539s 2018-11-29 21:08:09,887 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:08:09,887 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:08:09,964 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:10,147 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.2600698471069336s 2018-11-29 21:08:10,151 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:10,368 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:10,838 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6868960857391357s 2018-11-29 21:08:10,858 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:08:10,858 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:08:10,967 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:08:11,016 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:08:11,016 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:08:11,016 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:08:11,028 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318734 for request 300-0000331925 2018-11-29 21:08:11,035 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:08:11,035 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:08:11,036 INFO nodepool.NodeLauncher-0000318734: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318734 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318734 2018-11-29 21:08:11,037 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:08:11,238 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2741847038269043s 2018-11-29 21:08:11,354 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.986196756362915s 2018-11-29 21:08:11,842 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.8046789169311523s 2018-11-29 21:08:11,842 ERROR nodepool.NodeLauncher-0000318734: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318734: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:08:11,842 INFO nodepool.NodeLauncher-0000318734: Quota exceeded, invalidating quota cache 2018-11-29 21:08:11,843 INFO nodepool.NodeLauncher-0000318734: Aborting node 0000318734 due to quota failure 2018-11-29 21:08:12,506 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318733 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:08:12,506 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:08:12,512 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318733, state=deleting, external_id=None 2018-11-29 21:08:12,561 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000242494 (state: deleting, allocated_to: None) 2018-11-29 21:08:12,561 INFO nodepool.DeletedNodeWorker: Deleting deleting instance d0021aac-4e31-42c5-a6a6-b71097532a56 from rdo-cloud-tripleo 2018-11-29 21:08:12,566 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server d0021aac-4e31-42c5-a6a6-b71097532a56 2018-11-29 21:08:12,567 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:08:12,640 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.07289862632751465s 2018-11-29 21:08:13,101 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:08:14,367 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2660903930664062s 2018-11-29 21:08:16,296 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:16,687 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:17,327 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.640427827835083s 2018-11-29 21:08:17,669 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:08:17,671 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:08:17,672 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:08:17,673 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:08:17,673 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:08:17,674 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:08:17,674 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:08:17,675 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:08:17,675 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:08:17,675 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:08:17,676 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:08:17,676 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:08:17,677 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:08:17,678 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:08:17,765 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.468735933303833s 2018-11-29 21:08:20,396 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:08:21,040 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:08:21,272 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.23178458213806152s 2018-11-29 21:08:21,272 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:08:21,273 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:08:21,485 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.2116556167602539s 2018-11-29 21:08:21,488 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:21,546 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1500239372253418s 2018-11-29 21:08:21,888 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.399794340133667s 2018-11-29 21:08:21,908 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:08:21,908 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:08:22,020 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:08:22,075 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:08:22,075 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:08:22,075 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:08:22,085 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318735 for request 300-0000331819 2018-11-29 21:08:22,094 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:08:22,094 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:08:22,095 INFO nodepool.NodeLauncher-0000318735: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318735 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318735 2018-11-29 21:08:22,096 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:08:22,872 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:23,088 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318734 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:08:23,089 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:08:23,092 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318734, state=deleting, external_id=None 2018-11-29 21:08:23,343 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:23,805 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.7083280086517334s 2018-11-29 21:08:23,805 ERROR nodepool.NodeLauncher-0000318735: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318735: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:08:23,805 INFO nodepool.NodeLauncher-0000318735: Quota exceeded, invalidating quota cache 2018-11-29 21:08:23,806 INFO nodepool.NodeLauncher-0000318735: Aborting node 0000318735 due to quota failure 2018-11-29 21:08:24,060 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7173364162445068s 2018-11-29 21:08:24,086 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2139434814453125s 2018-11-29 21:08:27,579 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:08:28,468 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:08:28,475 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:08:28,476 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:08:28,476 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:08:28,477 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:08:28,477 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:08:28,477 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:08:28,478 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:08:28,478 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:08:28,479 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:08:28,479 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:08:28,482 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:08:28,482 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:08:28,483 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:08:28,826 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2466070652008057s 2018-11-29 21:08:29,217 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:29,388 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:30,084 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6959850788116455s 2018-11-29 21:08:30,131 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9135000705718994s 2018-11-29 21:08:30,529 ERROR nodepool.NodeDeleter: Exception deleting instance 75e44b32-409c-43fd-9ec9-fa590b96b917 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 75e44b32-409c-43fd-9ec9-fa590b96b917 deletion 2018-11-29 21:08:30,600 ERROR nodepool.NodeDeleter: Exception deleting instance 33034fa9-59e6-48d7-88ee-cd4f0bb01e35 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 33034fa9-59e6-48d7-88ee-cd4f0bb01e35 deletion 2018-11-29 21:08:32,099 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:08:32,252 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.15349483489990234s 2018-11-29 21:08:32,253 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:08:32,253 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:08:32,493 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.23946213722229004s 2018-11-29 21:08:32,496 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:33,223 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.7260382175445557s 2018-11-29 21:08:33,241 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:08:33,241 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:08:33,337 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:08:33,385 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:08:33,385 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:08:33,386 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:08:33,396 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318736 for request 300-0000331925 2018-11-29 21:08:33,405 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:08:33,406 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:08:33,407 INFO nodepool.NodeLauncher-0000318736: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318736 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318736 2018-11-29 21:08:33,408 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:08:33,738 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211297 (state: deleting, allocated_to: None) 2018-11-29 21:08:33,738 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 75e44b32-409c-43fd-9ec9-fa590b96b917 from rdo-cloud-tripleo 2018-11-29 21:08:33,744 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 75e44b32-409c-43fd-9ec9-fa590b96b917 2018-11-29 21:08:33,745 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:08:33,759 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211288 (state: deleting, allocated_to: None) 2018-11-29 21:08:33,759 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 33034fa9-59e6-48d7-88ee-cd4f0bb01e35 from rdo-cloud-tripleo 2018-11-29 21:08:33,764 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 33034fa9-59e6-48d7-88ee-cd4f0bb01e35 2018-11-29 21:08:33,815 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.0701749324798584s 2018-11-29 21:08:33,817 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:08:33,864 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318735 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:08:33,864 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:08:33,867 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318735, state=deleting, external_id=None 2018-11-29 21:08:34,021 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.20377397537231445s 2018-11-29 21:08:34,810 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4020459651947021s 2018-11-29 21:08:34,810 ERROR nodepool.NodeLauncher-0000318736: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318736: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:08:34,812 INFO nodepool.NodeLauncher-0000318736: Quota exceeded, invalidating quota cache 2018-11-29 21:08:34,812 INFO nodepool.NodeLauncher-0000318736: Aborting node 0000318736 due to quota failure 2018-11-29 21:08:34,853 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:08:36,085 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:36,147 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2933154106140137s 2018-11-29 21:08:36,160 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:36,719 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6333684921264648s 2018-11-29 21:08:37,096 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9307889938354492s 2018-11-29 21:08:39,195 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:08:39,196 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:08:39,197 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:08:39,198 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:08:39,198 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:08:39,198 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:08:39,199 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:08:39,199 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:08:39,200 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:08:39,200 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:08:39,200 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:08:39,203 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:08:39,203 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:08:39,203 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:08:40,165 ERROR nodepool.NodeDeleter: Exception deleting instance 3f4f9477-ce7a-4bd1-af21-8ce7eedc74e8 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 3f4f9477-ce7a-4bd1-af21-8ce7eedc74e8 deletion 2018-11-29 21:08:41,114 ERROR nodepool.NodeDeleter: Exception deleting instance 6a0c9d49-2789-4188-92f2-aa94b2491ce5 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 6a0c9d49-2789-4188-92f2-aa94b2491ce5 deletion 2018-11-29 21:08:42,105 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:42,120 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:42,841 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7355966567993164s 2018-11-29 21:08:43,134 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.013984203338623s 2018-11-29 21:08:43,148 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:08:43,410 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:08:43,557 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.1464531421661377s 2018-11-29 21:08:43,557 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:08:43,558 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:08:43,729 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.17084288597106934s 2018-11-29 21:08:43,732 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:44,197 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.4647519588470459s 2018-11-29 21:08:44,215 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:08:44,216 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:08:44,299 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1510131359100342s 2018-11-29 21:08:44,350 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:08:44,411 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:08:44,411 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:08:44,411 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:08:44,421 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318737 for request 300-0000331819 2018-11-29 21:08:44,432 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:08:44,432 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:08:44,434 INFO nodepool.NodeLauncher-0000318737: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318737 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318737 2018-11-29 21:08:44,434 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:08:44,459 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211294 (state: deleting, allocated_to: None) 2018-11-29 21:08:44,459 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 6a0c9d49-2789-4188-92f2-aa94b2491ce5 from rdo-cloud-tripleo 2018-11-29 21:08:44,465 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 6a0c9d49-2789-4188-92f2-aa94b2491ce5 2018-11-29 21:08:44,465 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:08:44,508 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211289 (state: deleting, allocated_to: None) 2018-11-29 21:08:44,508 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 3f4f9477-ce7a-4bd1-af21-8ce7eedc74e8 from rdo-cloud-tripleo 2018-11-29 21:08:44,514 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 3f4f9477-ce7a-4bd1-af21-8ce7eedc74e8 2018-11-29 21:08:44,539 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.07366275787353516s 2018-11-29 21:08:44,541 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:08:44,593 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318736 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:08:44,593 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:08:44,596 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318736, state=deleting, external_id=None 2018-11-29 21:08:44,819 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.27724456787109375s 2018-11-29 21:08:45,829 ERROR nodepool.NodeDeleter: Exception deleting instance b4853e6a-3115-451a-96cb-3c751068e393 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server b4853e6a-3115-451a-96cb-3c751068e393 deletion 2018-11-29 21:08:45,871 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4366254806518555s 2018-11-29 21:08:45,872 ERROR nodepool.NodeLauncher-0000318737: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318737: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:08:45,873 INFO nodepool.NodeLauncher-0000318737: Quota exceeded, invalidating quota cache 2018-11-29 21:08:45,873 INFO nodepool.NodeLauncher-0000318737: Aborting node 0000318737 due to quota failure 2018-11-29 21:08:48,326 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:48,742 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:49,888 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:08:49,891 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:08:49,894 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:08:49,895 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:08:49,895 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:08:49,896 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:08:49,896 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:08:49,896 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:08:49,897 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:08:49,897 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:08:49,898 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:08:49,898 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:08:49,899 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:08:49,902 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:08:49,978 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211299 (state: deleting, allocated_to: None) 2018-11-29 21:08:49,978 INFO nodepool.DeletedNodeWorker: Deleting deleting instance b4853e6a-3115-451a-96cb-3c751068e393 from rdo-cloud-tripleo 2018-11-29 21:08:49,983 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server b4853e6a-3115-451a-96cb-3c751068e393 2018-11-29 21:08:50,819 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 2.0761163234710693s 2018-11-29 21:08:51,193 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 2.8671724796295166s 2018-11-29 21:08:51,207 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 1) 2018-11-29 21:08:51,278 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.07039809226989746s 2018-11-29 21:08:51,280 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:08:52,607 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.326848030090332s 2018-11-29 21:08:54,464 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:08:54,601 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.13675618171691895s 2018-11-29 21:08:54,602 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:08:54,602 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:08:54,799 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.19733095169067383s 2018-11-29 21:08:54,803 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:55,519 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.7164924144744873s 2018-11-29 21:08:55,538 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:08:55,539 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:08:55,642 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:08:55,686 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:08:55,687 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:08:55,687 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:08:55,697 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318738 for request 300-0000331925 2018-11-29 21:08:55,705 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:08:55,706 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:08:55,707 INFO nodepool.NodeLauncher-0000318738: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318738 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318738 2018-11-29 21:08:55,708 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:08:56,540 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:56,833 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:57,034 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.3266665935516357s 2018-11-29 21:08:57,035 ERROR nodepool.NodeLauncher-0000318738: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318738: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:08:57,036 INFO nodepool.NodeLauncher-0000318738: Quota exceeded, invalidating quota cache 2018-11-29 21:08:57,036 INFO nodepool.NodeLauncher-0000318738: Aborting node 0000318738 due to quota failure 2018-11-29 21:08:57,432 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8918290138244629s 2018-11-29 21:08:57,601 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7679047584533691s 2018-11-29 21:08:57,735 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:08:58,177 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.4417119026184082s 2018-11-29 21:08:58,178 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-9ccf6f5b-0e2e-4f2f-8dea-89f549046748) 2018-11-29 21:08:58,324 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:08:58,635 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:08:59,979 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3439412117004395s 2018-11-29 21:09:00,667 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:09:00,668 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Declining node request 200-0000331938 because node type(s) [runc-centos] not available 2018-11-29 21:09:00,676 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:09:00,677 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:09:00,677 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:09:00,677 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:09:00,677 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:09:00,678 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:09:00,678 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:09:00,679 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:09:00,680 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:09:00,680 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:09:00,681 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:09:00,682 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:09:00,682 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:09:00,713 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 200-0000331938 2018-11-29 21:09:00,713 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:09:00,789 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318737 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:09:00,789 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:09:00,794 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318737, state=deleting, external_id=None 2018-11-29 21:09:02,572 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:02,871 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:03,527 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.954169511795044s 2018-11-29 21:09:03,624 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7534666061401367s 2018-11-29 21:09:05,710 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:09:06,005 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.29451680183410645s 2018-11-29 21:09:06,006 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:09:06,006 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:09:06,012 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:09:06,281 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.27463674545288086s 2018-11-29 21:09:06,284 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:06,768 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.48366332054138184s 2018-11-29 21:09:06,786 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:09:06,786 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:09:06,941 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:09:07,004 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:09:07,004 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:09:07,004 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:09:07,016 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318739 for request 300-0000331819 2018-11-29 21:09:07,023 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:09:07,023 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:09:07,025 INFO nodepool.NodeLauncher-0000318739: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318739 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318739 2018-11-29 21:09:07,025 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:09:07,327 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3156321048736572s 2018-11-29 21:09:08,730 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:09,323 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 2.2978241443634033s 2018-11-29 21:09:09,323 ERROR nodepool.NodeLauncher-0000318739: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318739: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:09:09,324 INFO nodepool.NodeLauncher-0000318739: Quota exceeded, invalidating quota cache 2018-11-29 21:09:09,324 INFO nodepool.NodeLauncher-0000318739: Aborting node 0000318739 due to quota failure 2018-11-29 21:09:09,623 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:09,638 ERROR nodepool.NodeDeleter: Exception deleting instance 9c02645e-baeb-4e9e-8a50-62d4671f8c9c from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 9c02645e-baeb-4e9e-8a50-62d4671f8c9c deletion 2018-11-29 21:09:09,983 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2519700527191162s 2018-11-29 21:09:10,286 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6621809005737305s 2018-11-29 21:09:11,364 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:09:11,365 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Declining node request 200-0000331938 because node type(s) [runc-centos] not available 2018-11-29 21:09:11,419 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:09:11,420 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Declining node request 300-0000331939 because node type(s) [rdo-centos-7] not available 2018-11-29 21:09:11,467 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Assigning node request 2018-11-29 21:09:11,468 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.small]: Declining node request 300-0000331940 because node type(s) [rdo-centos-7] not available 2018-11-29 21:09:11,468 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:09:11,469 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:09:11,472 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:09:11,474 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:09:11,475 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:09:11,475 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:09:11,476 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:09:11,476 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:09:11,476 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:09:11,477 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:09:11,477 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:09:11,516 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 200-0000331938 2018-11-29 21:09:11,517 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:09:11,519 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Removing handler for request 300-0000331940 2018-11-29 21:09:11,519 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:09:11,521 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 300-0000331939 2018-11-29 21:09:11,522 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:09:11,617 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318738 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:09:11,617 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:09:11,621 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318738, state=deleting, external_id=None 2018-11-29 21:09:11,698 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000242496 (state: deleting, allocated_to: None) 2018-11-29 21:09:11,698 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 9c02645e-baeb-4e9e-8a50-62d4671f8c9c from rdo-cloud-tripleo 2018-11-29 21:09:11,703 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 9c02645e-baeb-4e9e-8a50-62d4671f8c9c 2018-11-29 21:09:11,704 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:09:11,774 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.06958746910095215s 2018-11-29 21:09:13,356 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:09:14,583 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2263329029083252s 2018-11-29 21:09:15,200 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:15,647 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:16,406 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2062232494354248s 2018-11-29 21:09:16,479 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8321497440338135s 2018-11-29 21:09:17,033 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:09:17,196 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.1628730297088623s 2018-11-29 21:09:17,197 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:09:17,197 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:09:17,390 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.19264817237854004s 2018-11-29 21:09:17,394 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:17,416 ERROR nodepool.NodeDeleter: Exception deleting instance 0f13ac4c-c10c-4892-8172-561e6f949d19 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 0f13ac4c-c10c-4892-8172-561e6f949d19 deletion 2018-11-29 21:09:18,176 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.7818603515625s 2018-11-29 21:09:18,194 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:09:18,194 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:09:18,303 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:09:18,354 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:09:18,354 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:09:18,354 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:09:18,364 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318740 for request 300-0000331925 2018-11-29 21:09:18,375 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:09:18,375 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:09:18,377 INFO nodepool.NodeLauncher-0000318740: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318740 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318740 2018-11-29 21:09:18,378 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:09:19,727 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.3483643531799316s 2018-11-29 21:09:19,727 ERROR nodepool.NodeLauncher-0000318740: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318740: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:09:19,729 INFO nodepool.NodeLauncher-0000318740: Quota exceeded, invalidating quota cache 2018-11-29 21:09:19,729 INFO nodepool.NodeLauncher-0000318740: Aborting node 0000318740 due to quota failure 2018-11-29 21:09:20,613 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:09:21,876 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2618513107299805s 2018-11-29 21:09:21,897 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:22,151 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Assigning node request 2018-11-29 21:09:22,153 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.small]: Declining node request 200-0000331938 because node type(s) [runc-centos] not available 2018-11-29 21:09:22,189 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:09:22,215 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetLimits (queue 0) 2018-11-29 21:09:22,218 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:09:22,219 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:09:22,219 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Declining node request 300-0000331940 because node type(s) [rdo-centos-7] not available 2018-11-29 21:09:22,219 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:09:22,220 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:09:22,220 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:09:22,223 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:09:22,226 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:09:22,227 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:09:22,228 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:09:22,229 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:09:22,231 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:09:22,231 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:09:22,244 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211293 (state: deleting, allocated_to: None) 2018-11-29 21:09:22,244 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 0f13ac4c-c10c-4892-8172-561e6f949d19 from rdo-cloud-tripleo 2018-11-29 21:09:22,249 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 0f13ac4c-c10c-4892-8172-561e6f949d19 2018-11-29 21:09:22,263 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetLimits in 0.048119306564331055s 2018-11-29 21:09:22,263 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for rdo-cloud: {'compute': {'cores': 520, 'ram': 532480, 'instances': 65}} 2018-11-29 21:09:22,264 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:09:22,270 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 300-0000331940 2018-11-29 21:09:22,270 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:09:22,275 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Assigning node request 2018-11-29 21:09:22,275 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.small]: Declining node request 300-0000331940 because node type(s) [rdo-centos-7] not available 2018-11-29 21:09:22,327 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetFlavorsDetail in 0.0630636215209961s 2018-11-29 21:09:22,327 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:22,332 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for rdo-cloud: {'compute': {'cores': 519, 'ram': 530432, 'instances': 64}} 2018-11-29 21:09:22,332 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Accepting node request 300-0000331939 2018-11-29 21:09:22,353 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Removing handler for request 200-0000331938 2018-11-29 21:09:22,353 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Removing handler for request 300-0000331940 2018-11-29 21:09:22,353 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:09:22,423 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318739 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:09:22,423 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:09:22,430 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318739, state=deleting, external_id=None 2018-11-29 21:09:22,519 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Locked existing node 0000318552 for request 300-0000331939 2018-11-29 21:09:22,535 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:09:22,539 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Accepting node request 300-0000331940 2018-11-29 21:09:22,653 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:09:22,672 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:09:22,681 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Predicted remaining provider quota: {'compute': {'cores': 431, 'ram': 399360, 'instances': 48}} 2018-11-29 21:09:22,767 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 28}} 2018-11-29 21:09:22,767 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 27}} 2018-11-29 21:09:22,779 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Locked building node 0000318741 for request 300-0000331940 2018-11-29 21:09:22,783 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Fulfilled node request 300-0000331939 2018-11-29 21:09:22,786 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Unlocked node 0000318552 for request 300-0000331939 2018-11-29 21:09:22,790 INFO nodepool.NodeLauncher-0000318741: Creating server with hostname rdo-centos-7-rdo-cloud-0000318741 in rdo-cloud from image rdo-centos-7 for node id: 0000318741 2018-11-29 21:09:22,791 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 300-0000331939 2018-11-29 21:09:22,792 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: ['300-0000331940'] 2018-11-29 21:09:23,131 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2342619895935059s 2018-11-29 21:09:23,142 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:09:23,220 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.07749438285827637s 2018-11-29 21:09:23,306 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.978522539138794s 2018-11-29 21:09:23,306 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputePostServers (queue 0) 2018-11-29 21:09:24,431 ERROR nodepool.NodeDeleter: Exception deleting instance 52af0012-6653-4e5d-adcf-291841fe4c5f from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 52af0012-6653-4e5d-adcf-291841fe4c5f deletion 2018-11-29 21:09:25,383 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputePostServers in 2.076446771621704s 2018-11-29 21:09:25,383 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServers (queue 0) 2018-11-29 21:09:25,475 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServers in 0.09165573120117188s 2018-11-29 21:09:25,480 DEBUG nodepool.NodeLauncher-0000318741: Waiting for server 5960ec0b-7a59-4481-8d7e-942c1336baf7 for node id: 0000318741 2018-11-29 21:09:27,623 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211295 (state: deleting, allocated_to: None) 2018-11-29 21:09:27,623 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 52af0012-6653-4e5d-adcf-291841fe4c5f from rdo-cloud-tripleo 2018-11-29 21:09:27,630 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 52af0012-6653-4e5d-adcf-291841fe4c5f 2018-11-29 21:09:27,631 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:09:27,704 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.0729365348815918s 2018-11-29 21:09:27,907 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:09:28,380 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:09:28,501 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:28,520 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.13979816436767578s 2018-11-29 21:09:28,521 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:09:28,521 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:09:28,773 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.25204896926879883s 2018-11-29 21:09:28,776 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:29,140 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6393711566925049s 2018-11-29 21:09:29,189 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.41205286979675293s 2018-11-29 21:09:29,206 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:09:29,206 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:09:29,311 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:09:29,368 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4593348503112793s 2018-11-29 21:09:29,387 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:29,394 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:09:29,394 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:09:29,394 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:09:29,405 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318742 for request 300-0000331819 2018-11-29 21:09:29,412 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:09:29,412 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:09:29,413 INFO nodepool.NodeLauncher-0000318742: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318742 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318742 2018-11-29 21:09:29,414 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:09:30,612 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.198366641998291s 2018-11-29 21:09:30,613 ERROR nodepool.NodeLauncher-0000318742: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318742: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:09:30,614 INFO nodepool.NodeLauncher-0000318742: Quota exceeded, invalidating quota cache 2018-11-29 21:09:30,614 INFO nodepool.NodeLauncher-0000318742: Aborting node 0000318742 due to quota failure 2018-11-29 21:09:30,627 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2396345138549805s 2018-11-29 21:09:31,149 ERROR nodepool.NodeDeleter: Exception deleting instance e5999063-a1b8-412a-bd0e-ef26c58f0381 from rdo-cloud: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server e5999063-a1b8-412a-bd0e-ef26c58f0381 deletion 2018-11-29 21:09:32,651 INFO nodepool.NodePool: Creating requests for 1 rdo-centos-7 nodes 2018-11-29 21:09:32,796 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Assigning node request 2018-11-29 21:09:32,798 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.large]: Declining node request 200-0000331938 because node type(s) [runc-centos] not available 2018-11-29 21:09:32,804 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:09:32,810 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:09:32,814 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:09:32,820 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:09:32,827 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:09:32,830 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:09:32,831 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:09:32,832 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:09:32,833 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:09:32,834 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:09:32,840 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:09:32,851 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Removing handler for request 200-0000331938 2018-11-29 21:09:32,852 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:09:32,864 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:09:33,002 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:09:33,017 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000260472 (state: deleting, allocated_to: None) 2018-11-29 21:09:33,017 INFO nodepool.DeletedNodeWorker: Deleting deleting instance e5999063-a1b8-412a-bd0e-ef26c58f0381 from rdo-cloud 2018-11-29 21:09:33,022 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server e5999063-a1b8-412a-bd0e-ef26c58f0381 2018-11-29 21:09:33,023 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeDeleteServers (queue 0) 2018-11-29 21:09:33,032 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318740 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:09:33,032 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:09:33,035 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318740, state=deleting, external_id=None 2018-11-29 21:09:33,090 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeDeleteServers in 0.06687188148498535s 2018-11-29 21:09:33,180 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Accepting node request 100-0000331941 2018-11-29 21:09:33,281 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:09:33,297 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:09:33,305 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Predicted remaining provider quota: {'compute': {'cores': 423, 'ram': 391168, 'instances': 47}} 2018-11-29 21:09:33,372 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 27}} 2018-11-29 21:09:33,372 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 26}} 2018-11-29 21:09:33,384 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Locked building node 0000318743 for request 100-0000331941 2018-11-29 21:09:33,396 INFO nodepool.NodeLauncher-0000318743: Creating server with hostname rdo-centos-7-rdo-cloud-0000318743 in rdo-cloud from image rdo-centos-7 for node id: 0000318743 2018-11-29 21:09:33,396 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputePostServers (queue 0) 2018-11-29 21:09:33,399 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:09:33,399 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Declining node request 200-0000331938 because node type(s) [runc-centos] not available 2018-11-29 21:09:33,415 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 200-0000331938 2018-11-29 21:09:33,415 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: ['300-0000331940', '100-0000331941'] 2018-11-29 21:09:35,130 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputePostServers in 1.733884334564209s 2018-11-29 21:09:35,131 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 1) 2018-11-29 21:09:35,399 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:09:35,952 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8205504417419434s 2018-11-29 21:09:35,960 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServers (queue 0) 2018-11-29 21:09:36,474 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServers in 0.5133440494537354s 2018-11-29 21:09:36,480 DEBUG nodepool.NodeLauncher-0000318743: Waiting for server 0f348152-e2c3-45de-b323-3c188ef9f4da for node id: 0000318743 2018-11-29 21:09:36,849 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4500138759613037s 2018-11-29 21:09:36,865 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:37,952 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0868144035339355s 2018-11-29 21:09:39,417 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:09:39,512 ERROR nodepool.NodeDeleter: Exception deleting instance bbbdbcbc-cfa9-4ee1-828e-03a90bdda5ac from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server bbbdbcbc-cfa9-4ee1-828e-03a90bdda5ac deletion 2018-11-29 21:09:39,929 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.5118799209594727s 2018-11-29 21:09:39,930 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:09:39,930 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:09:40,127 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1964402198791504s 2018-11-29 21:09:40,130 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:40,494 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetPorts (queue 0) 2018-11-29 21:09:40,523 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.39284610748291016s 2018-11-29 21:09:40,542 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:09:40,542 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:09:40,642 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:09:40,688 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:09:40,689 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:09:40,689 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:09:40,849 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetPorts in 0.3547792434692383s 2018-11-29 21:09:40,855 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:09:40,930 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.07510495185852051s 2018-11-29 21:09:40,955 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkPostFloatingips (queue 0) 2018-11-29 21:09:41,702 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318744 for request 300-0000331925 2018-11-29 21:09:41,710 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:09:41,710 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:09:41,712 INFO nodepool.NodeLauncher-0000318744: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318744 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318744 2018-11-29 21:09:41,713 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:09:42,881 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:09:43,135 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4220473766326904s 2018-11-29 21:09:43,135 ERROR nodepool.NodeLauncher-0000318744: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318744: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:09:43,136 INFO nodepool.NodeLauncher-0000318744: Quota exceeded, invalidating quota cache 2018-11-29 21:09:43,136 INFO nodepool.NodeLauncher-0000318744: Aborting node 0000318744 due to quota failure 2018-11-29 21:09:43,387 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:09:43,411 INFO nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Assigning node request 2018-11-29 21:09:43,412 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-tripleo-ovb]: Declining node request 200-0000331938 because node type(s) [runc-centos] not available 2018-11-29 21:09:43,452 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:09:43,483 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:09:43,513 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:09:43,515 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:09:43,517 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Assigning node request 2018-11-29 21:09:43,517 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.large]: Declining node request 200-0000331938 because node type(s) [runc-centos] not available 2018-11-29 21:09:43,522 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Removing handler for request 200-0000331938 2018-11-29 21:09:43,523 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:09:43,523 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:09:43,526 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:09:43,529 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:09:43,532 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:09:43,532 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:09:43,534 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkPostFloatingips in 2.5789294242858887s 2018-11-29 21:09:43,538 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:43,541 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:09:43,580 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Removing handler for request 200-0000331938 2018-11-29 21:09:43,580 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:09:43,745 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: ['300-0000331940', '100-0000331941'] 2018-11-29 21:09:43,750 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318742 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:09:43,750 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:09:43,755 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318742, state=deleting, external_id=None 2018-11-29 21:09:43,796 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000242421 (state: deleting, allocated_to: None) 2018-11-29 21:09:43,796 INFO nodepool.DeletedNodeWorker: Deleting deleting instance bbbdbcbc-cfa9-4ee1-828e-03a90bdda5ac from rdo-cloud-tripleo 2018-11-29 21:09:43,802 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server bbbdbcbc-cfa9-4ee1-828e-03a90bdda5ac 2018-11-29 21:09:44,379 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4978723526000977s 2018-11-29 21:09:44,395 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 1) 2018-11-29 21:09:44,491 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9530320167541504s 2018-11-29 21:09:46,493 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetPorts (queue 0) 2018-11-29 21:09:46,635 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 2.2385895252227783s 2018-11-29 21:09:46,635 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:09:46,713 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetPorts in 0.21933507919311523s 2018-11-29 21:09:46,718 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:09:46,740 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.10499072074890137s 2018-11-29 21:09:46,784 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.06663370132446289s 2018-11-29 21:09:46,809 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkPostFloatingips (queue 0) 2018-11-29 21:09:48,550 DEBUG nodepool.NodeLauncher-0000318741: Node 0000318741 is running [region: regionOne, az: nova, ip: 38.145.34.40 ipv4: 38.145.34.40, ipv6: ] 2018-11-29 21:09:48,550 DEBUG nodepool.NodeLauncher-0000318741: Gathering host keys for node 0000318741 2018-11-29 21:09:48,588 INFO nodepool.NodeLauncher-0000318741: Node id 0000318741 is ready 2018-11-29 21:09:49,065 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318700 (state: used, allocated_to: 100-0000331934) 2018-11-29 21:09:49,065 INFO nodepool.DeletedNodeWorker: Deleting used instance None from ansible-statics 2018-11-29 21:09:49,072 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318700, state=deleting, external_id=None 2018-11-29 21:09:49,091 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkPostFloatingips in 2.2819571495056152s 2018-11-29 21:09:49,485 DEBUG nodepool.driver.static.StaticNodeProvider: Registered static node 35.236.232.200 2018-11-29 21:09:49,988 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:50,408 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:09:51,716 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:09:51,842 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.8524670600891113s 2018-11-29 21:09:51,850 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.13414216041564941s 2018-11-29 21:09:51,851 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:09:51,851 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:09:52,014 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.16254067420959473s 2018-11-29 21:09:52,017 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:52,060 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.6519451141357422s 2018-11-29 21:09:52,071 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:52,324 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.30678439140319824s 2018-11-29 21:09:52,341 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:09:52,341 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:09:52,454 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:09:52,504 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:09:52,504 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:09:52,504 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:09:52,514 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318746 for request 300-0000331819 2018-11-29 21:09:52,523 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:09:52,524 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:09:52,525 INFO nodepool.NodeLauncher-0000318746: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318746 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318746 2018-11-29 21:09:52,526 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:09:53,418 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.8919563293457031s 2018-11-29 21:09:53,419 ERROR nodepool.NodeLauncher-0000318746: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318746: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:09:53,419 INFO nodepool.NodeLauncher-0000318746: Quota exceeded, invalidating quota cache 2018-11-29 21:09:53,419 INFO nodepool.NodeLauncher-0000318746: Aborting node 0000318746 due to quota failure 2018-11-29 21:09:53,466 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.394754409790039s 2018-11-29 21:09:53,505 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:09:53,506 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Declining node request 200-0000331942 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:09:53,562 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:09:53,563 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Declining node request 200-0000331943 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:09:53,649 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 200-0000331942 2018-11-29 21:09:53,649 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 200-0000331943 2018-11-29 21:09:53,649 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:09:53,914 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 21:09:53,914 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 200-0000331938 because node type(s) [runc-centos] not available 2018-11-29 21:09:54,000 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:09:54,000 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 200-0000331938 because node type(s) [runc-centos] not available 2018-11-29 21:09:54,012 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 21:09:54,012 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 200-0000331942 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:09:54,100 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:09:54,105 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:09:54,105 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 200-0000331942 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:09:54,110 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Assigning node request 2018-11-29 21:09:54,113 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1]: Declining node request 200-0000331938 because node type(s) [runc-centos] not available 2018-11-29 21:09:54,116 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 21:09:54,117 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 200-0000331943 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:09:54,153 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:09:54,180 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.07911872863769531s 2018-11-29 21:09:54,209 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetPorts (queue 0) 2018-11-29 21:09:54,223 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:09:54,233 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:09:54,233 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:09:54,235 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:09:54,247 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:09:54,247 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:09:54,247 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:09:54,248 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Assigning node request 2018-11-29 21:09:54,248 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.large]: Declining node request 200-0000331942 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:09:54,251 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 200-0000331938 2018-11-29 21:09:54,251 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 200-0000331942 2018-11-29 21:09:54,251 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 200-0000331943 2018-11-29 21:09:54,251 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:09:54,253 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:09:54,254 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 200-0000331943 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:09:54,277 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Removing handler for request 200-0000331938 2018-11-29 21:09:54,277 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:09:54,302 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Assigning node request 2018-11-29 21:09:54,303 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.large]: Declining node request 200-0000331943 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:09:54,304 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 200-0000331938 2018-11-29 21:09:54,304 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 200-0000331942 2018-11-29 21:09:54,304 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 200-0000331943 2018-11-29 21:09:54,304 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:09:54,332 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:09:54,332 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Declining node request 200-0000331942 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:09:54,343 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Removing handler for request 200-0000331942 2018-11-29 21:09:54,344 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Removing handler for request 200-0000331943 2018-11-29 21:09:54,344 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:09:54,381 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:09:54,381 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Declining node request 200-0000331943 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:09:54,413 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetPorts in 0.20431113243103027s 2018-11-29 21:09:54,420 DEBUG nodepool.NodeLauncher-0000318743: Node 0000318743 is running [region: regionOne, az: nova, ip: 38.145.32.87 ipv4: 38.145.32.87, ipv6: ] 2018-11-29 21:09:54,420 DEBUG nodepool.NodeLauncher-0000318743: Gathering host keys for node 0000318743 2018-11-29 21:09:54,421 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Fulfilled node request 300-0000331940 2018-11-29 21:09:54,425 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Unlocked node 0000318741 for request 300-0000331940 2018-11-29 21:09:54,435 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 300-0000331940 2018-11-29 21:09:54,435 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 200-0000331942 2018-11-29 21:09:54,435 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 200-0000331943 2018-11-29 21:09:54,435 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: ['100-0000331941'] 2018-11-29 21:09:54,458 INFO nodepool.NodeLauncher-0000318743: Node id 0000318743 is ready 2018-11-29 21:09:54,514 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318744 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:09:54,514 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:09:54,518 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318744, state=deleting, external_id=None 2018-11-29 21:09:57,857 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:58,091 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:09:58,603 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.745635986328125s 2018-11-29 21:09:58,737 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:09:59,181 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.44382405281066895s 2018-11-29 21:09:59,181 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-8562e670-943d-406f-9d55-d446baca277f) 2018-11-29 21:09:59,317 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:09:59,324 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2337617874145508s 2018-11-29 21:09:59,335 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:00,466 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1261012554168701s 2018-11-29 21:10:02,529 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:10:02,682 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.15281462669372559s 2018-11-29 21:10:02,682 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:10:02,683 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:10:02,892 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.20912909507751465s 2018-11-29 21:10:02,895 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:03,567 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.671959400177002s 2018-11-29 21:10:03,585 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:10:03,585 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:10:03,716 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:10:03,735 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:10:03,782 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:10:03,782 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:10:03,782 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:10:03,793 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318747 for request 300-0000331925 2018-11-29 21:10:03,799 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:10:03,799 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:10:03,801 INFO nodepool.NodeLauncher-0000318747: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318747 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318747 2018-11-29 21:10:03,802 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:10:04,230 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:10:04,233 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Accepting node request 200-0000331942 2018-11-29 21:10:04,524 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:04,683 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Locked existing node 0000318634 for request 200-0000331942 2018-11-29 21:10:04,689 INFO nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Assigning node request 2018-11-29 21:10:04,690 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-tripleo-ovb]: Declining node request 200-0000331938 because node type(s) [runc-centos] not available 2018-11-29 21:10:04,707 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:10:04,728 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Accepting node request 200-0000331943 2018-11-29 21:10:04,742 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:10:04,744 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:10:04,765 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:10:04,773 INFO nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Assigning node request 2018-11-29 21:10:04,774 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4]: Declining node request 200-0000331938 because node type(s) [runc-centos] not available 2018-11-29 21:10:04,777 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Removing handler for request 200-0000331938 2018-11-29 21:10:04,777 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:10:04,777 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:10:04,779 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:10:04,797 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:10:04,803 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:10:04,816 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:10:04,819 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Removing handler for request 200-0000331938 2018-11-29 21:10:04,819 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:10:04,828 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:10:04,841 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Fulfilled node request 100-0000331941 2018-11-29 21:10:04,843 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Unlocked node 0000318743 for request 100-0000331941 2018-11-29 21:10:04,849 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 100-0000331941 2018-11-29 21:10:04,849 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:10:04,974 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:10:05,023 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:10:05,037 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318746 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:10:05,037 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:10:05,040 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318746, state=deleting, external_id=None 2018-11-29 21:10:05,043 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Predicted remaining provider quota: {'compute': {'cores': 419, 'ram': 382976, 'instances': 46}} 2018-11-29 21:10:05,168 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 15}} 2018-11-29 21:10:05,168 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 14}} 2018-11-29 21:10:05,199 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Locked building node 0000318748 for request 200-0000331943 2018-11-29 21:10:05,204 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Fulfilled node request 200-0000331942 2018-11-29 21:10:05,207 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Unlocked node 0000318634 for request 200-0000331942 2018-11-29 21:10:05,207 INFO nodepool.NodeLauncher-0000318748: Creating server with hostname dib-dci-centos-7-rdo-cloud-0000318748 in rdo-cloud from image dib-dci-centos-7 for node id: 0000318748 2018-11-29 21:10:05,214 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 200-0000331942 2018-11-29 21:10:05,214 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: ['200-0000331943'] 2018-11-29 21:10:05,265 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4630346298217773s 2018-11-29 21:10:05,265 ERROR nodepool.NodeLauncher-0000318747: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318747: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:10:05,265 INFO nodepool.NodeLauncher-0000318747: Quota exceeded, invalidating quota cache 2018-11-29 21:10:05,266 INFO nodepool.NodeLauncher-0000318747: Aborting node 0000318747 due to quota failure 2018-11-29 21:10:05,359 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:10:05,578 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.0532467365264893s 2018-11-29 21:10:05,578 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputePostServers (queue 0) 2018-11-29 21:10:06,634 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2749559879302979s 2018-11-29 21:10:06,649 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:06,749 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputePostServers in 1.1708505153656006s 2018-11-29 21:10:06,749 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServers (queue 0) 2018-11-29 21:10:06,977 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServers in 0.2274165153503418s 2018-11-29 21:10:06,982 DEBUG nodepool.NodeLauncher-0000318748: Waiting for server 7779798d-36d4-40d1-9e91-29b211616bf8 for node id: 0000318748 2018-11-29 21:10:08,001 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.3501429557800293s 2018-11-29 21:10:10,626 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:11,331 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7047550678253174s 2018-11-29 21:10:12,663 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:10:13,712 DEBUG nodepool.NodePool: No more active min-ready requests for label rdo-centos-7 2018-11-29 21:10:13,780 INFO nodepool.NodePool: Creating requests for 1 dib-dci-centos-7 nodes 2018-11-29 21:10:13,805 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:10:13,819 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:10:14,023 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.21758437156677246s 2018-11-29 21:10:14,024 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:10:14,026 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:10:14,160 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1341381072998047s 2018-11-29 21:10:14,164 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:14,775 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 2.111754894256592s 2018-11-29 21:10:14,788 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:14,905 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.7414219379425049s 2018-11-29 21:10:14,924 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:10:14,924 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:10:15,188 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Assigning node request 2018-11-29 21:10:15,208 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:10:15,342 INFO nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Assigning node request 2018-11-29 21:10:15,343 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-rdo-ca-ymq-1-main]: Declining node request 200-0000331938 because node type(s) [runc-centos] not available 2018-11-29 21:10:15,401 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:10:15,403 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:10:15,403 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:10:15,404 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:10:15,409 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:10:15,428 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:10:15,435 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:10:15,441 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Removing handler for request 200-0000331938 2018-11-29 21:10:15,441 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:10:15,442 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:10:15,450 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:10:15,518 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.large]: Declining node request 100-0000331944 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:10:15,546 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Removing handler for request 100-0000331944 2018-11-29 21:10:15,547 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:10:15,547 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:10:15,615 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:10:15,765 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Accepting node request 100-0000331944 2018-11-29 21:10:15,816 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:10:15,816 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:10:15,816 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:10:15,827 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318749 for request 300-0000331819 2018-11-29 21:10:15,838 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:10:15,838 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:10:15,846 INFO nodepool.NodeLauncher-0000318749: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318749 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318749 2018-11-29 21:10:15,847 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:10:15,950 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:10:15,972 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:10:15,982 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Predicted remaining provider quota: {'compute': {'cores': 415, 'ram': 374784, 'instances': 45}} 2018-11-29 21:10:16,063 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 14}} 2018-11-29 21:10:16,063 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 13}} 2018-11-29 21:10:16,074 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Locked building node 0000318750 for request 100-0000331944 2018-11-29 21:10:16,080 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: ['200-0000331943', '100-0000331944'] 2018-11-29 21:10:16,080 INFO nodepool.NodeLauncher-0000318750: Creating server with hostname dib-dci-centos-7-rdo-cloud-0000318750 in rdo-cloud from image dib-dci-centos-7 for node id: 0000318750 2018-11-29 21:10:16,081 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputePostServers (queue 0) 2018-11-29 21:10:17,022 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 2.2291619777679443s 2018-11-29 21:10:17,122 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2753267288208008s 2018-11-29 21:10:17,123 ERROR nodepool.NodeLauncher-0000318749: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318749: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:10:17,125 INFO nodepool.NodeLauncher-0000318749: Quota exceeded, invalidating quota cache 2018-11-29 21:10:17,125 INFO nodepool.NodeLauncher-0000318749: Aborting node 0000318749 due to quota failure 2018-11-29 21:10:18,077 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputePostServers in 1.9960496425628662s 2018-11-29 21:10:18,078 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 1) 2018-11-29 21:10:18,504 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.42626190185546875s 2018-11-29 21:10:18,514 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServers (queue 1) 2018-11-29 21:10:18,909 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServers in 0.3951430320739746s 2018-11-29 21:10:18,912 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetPorts (queue 0) 2018-11-29 21:10:18,917 DEBUG nodepool.NodeLauncher-0000318750: Waiting for server 82795ae5-7aa5-4ab7-ac8a-6e5adde268d3 for node id: 0000318750 2018-11-29 21:10:19,242 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetPorts in 0.3295173645019531s 2018-11-29 21:10:19,247 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:10:19,312 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.06507039070129395s 2018-11-29 21:10:19,339 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkPostFloatingips (queue 0) 2018-11-29 21:10:20,808 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:10:21,788 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318747 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:10:21,788 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:10:21,793 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318747, state=deleting, external_id=None 2018-11-29 21:10:21,899 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkPostFloatingips in 2.559746026992798s 2018-11-29 21:10:22,177 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3692026138305664s 2018-11-29 21:10:22,193 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:23,484 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2876629829406738s 2018-11-29 21:10:23,618 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:23,990 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:10:25,175 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.5567126274108887s 2018-11-29 21:10:25,285 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:10:25,816 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:10:25,822 INFO nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Assigning node request 2018-11-29 21:10:25,823 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Accepting node request 200-0000331938 2018-11-29 21:10:25,843 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:10:25,844 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:10:25,855 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:10:25,860 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:10:25,878 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:10:25,889 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:10:25,895 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:10:25,896 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:10:25,897 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:10:25,915 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:10:25,955 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Locked existing node 0000318621 for request 200-0000331938 2018-11-29 21:10:25,962 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Fulfilled node request 200-0000331938 2018-11-29 21:10:25,972 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Unlocked node 0000318621 for request 200-0000331938 2018-11-29 21:10:25,979 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Removing handler for request 200-0000331938 2018-11-29 21:10:25,979 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:10:26,060 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.19969582557678223s 2018-11-29 21:10:26,060 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:10:26,061 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:10:26,196 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13532233238220215s 2018-11-29 21:10:26,200 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:26,250 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: ['200-0000331943', '100-0000331944'] 2018-11-29 21:10:26,814 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6142542362213135s 2018-11-29 21:10:26,833 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:10:26,833 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:10:26,907 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:10:27,068 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.16105866432189941s 2018-11-29 21:10:27,371 DEBUG nodepool.NodeLauncher-0000318748: Node 0000318748 is running [region: regionOne, az: nova, ip: 38.145.32.98 ipv4: 38.145.32.98, ipv6: ] 2018-11-29 21:10:27,371 DEBUG nodepool.NodeLauncher-0000318748: Gathering host keys for node 0000318748 2018-11-29 21:10:27,403 INFO nodepool.NodeLauncher-0000318748: Node id 0000318748 is ready 2018-11-29 21:10:27,618 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:10:27,721 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:10:27,721 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:10:27,721 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:10:27,731 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318751 for request 300-0000331925 2018-11-29 21:10:27,737 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:10:27,738 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:10:27,739 INFO nodepool.NodeLauncher-0000318751: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318751 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318751 2018-11-29 21:10:27,740 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:10:28,209 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:10:28,617 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.8774657249450684s 2018-11-29 21:10:28,618 ERROR nodepool.NodeLauncher-0000318751: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318751: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:10:28,618 INFO nodepool.NodeLauncher-0000318751: Quota exceeded, invalidating quota cache 2018-11-29 21:10:28,618 INFO nodepool.NodeLauncher-0000318751: Aborting node 0000318751 due to quota failure 2018-11-29 21:10:29,388 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1787760257720947s 2018-11-29 21:10:29,403 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:30,184 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.7784585952758789s 2018-11-29 21:10:31,191 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:31,407 ERROR nodepool.NodeDeleter: Exception deleting instance fbf7b562-5826-4b67-a10b-87b962a17f12 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server fbf7b562-5826-4b67-a10b-87b962a17f12 deletion 2018-11-29 21:10:32,110 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9181714057922363s 2018-11-29 21:10:32,769 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211298 (state: deleting, allocated_to: None) 2018-11-29 21:10:32,770 INFO nodepool.DeletedNodeWorker: Deleting deleting instance fbf7b562-5826-4b67-a10b-87b962a17f12 from rdo-cloud-tripleo 2018-11-29 21:10:32,776 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server fbf7b562-5826-4b67-a10b-87b962a17f12 2018-11-29 21:10:32,776 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:10:32,845 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.06883907318115234s 2018-11-29 21:10:32,906 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318749 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:10:32,906 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:10:32,909 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318749, state=deleting, external_id=None 2018-11-29 21:10:33,937 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetPorts (queue 0) 2018-11-29 21:10:34,147 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetPorts in 0.20999693870544434s 2018-11-29 21:10:34,152 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:10:34,210 INFO nodepool.NodePool: Creating requests for 1 runc-centos nodes 2018-11-29 21:10:34,216 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:10:34,408 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.2562384605407715s 2018-11-29 21:10:34,437 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkPostFloatingips (queue 0) 2018-11-29 21:10:34,852 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:10:35,371 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Assigning node request 2018-11-29 21:10:35,431 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.small]: Declining node request 100-0000331945 because node type(s) [runc-centos] not available 2018-11-29 21:10:35,588 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Removing handler for request 100-0000331945 2018-11-29 21:10:35,588 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:10:36,055 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Assigning node request 2018-11-29 21:10:36,186 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3343541622161865s 2018-11-29 21:10:36,212 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:36,281 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:10:36,291 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:10:36,331 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:10:36,399 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:10:36,429 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:10:36,437 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:10:36,439 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:10:36,443 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:10:36,451 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:10:36,475 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:10:36,478 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1]: Declining node request 100-0000331945 because node type(s) [runc-centos] not available 2018-11-29 21:10:36,495 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Removing handler for request 100-0000331945 2018-11-29 21:10:36,495 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:10:36,518 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:10:36,575 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Declining node request 100-0000331945 because node type(s) [runc-centos] not available 2018-11-29 21:10:36,591 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Fulfilled node request 200-0000331943 2018-11-29 21:10:36,595 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Unlocked node 0000318748 for request 200-0000331943 2018-11-29 21:10:36,603 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 200-0000331943 2018-11-29 21:10:36,604 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 100-0000331945 2018-11-29 21:10:36,604 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: ['100-0000331944'] 2018-11-29 21:10:36,994 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkPostFloatingips in 2.5569300651550293s 2018-11-29 21:10:37,122 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9095301628112793s 2018-11-29 21:10:37,376 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:37,745 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:10:37,915 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.16987967491149902s 2018-11-29 21:10:37,916 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:10:37,916 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:10:38,121 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.2052593231201172s 2018-11-29 21:10:38,125 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:38,284 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9076106548309326s 2018-11-29 21:10:38,522 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.3970777988433838s 2018-11-29 21:10:38,540 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:10:38,540 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:10:38,644 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:10:38,695 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:10:38,695 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:10:38,695 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:10:38,707 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318752 for request 300-0000331819 2018-11-29 21:10:38,715 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:10:38,715 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:10:38,717 INFO nodepool.NodeLauncher-0000318752: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318752 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318752 2018-11-29 21:10:38,717 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:10:40,048 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.3303024768829346s 2018-11-29 21:10:40,048 ERROR nodepool.NodeLauncher-0000318752: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318752: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:10:40,048 INFO nodepool.NodeLauncher-0000318752: Quota exceeded, invalidating quota cache 2018-11-29 21:10:40,049 INFO nodepool.NodeLauncher-0000318752: Aborting node 0000318752 due to quota failure 2018-11-29 21:10:42,004 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:10:42,084 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.07983183860778809s 2018-11-29 21:10:42,113 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetPorts (queue 0) 2018-11-29 21:10:42,157 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:42,326 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetPorts in 0.212050199508667s 2018-11-29 21:10:42,332 DEBUG nodepool.NodeLauncher-0000318750: Node 0000318750 is running [region: regionOne, az: nova, ip: 38.145.32.95 ipv4: 38.145.32.95, ipv6: ] 2018-11-29 21:10:42,332 DEBUG nodepool.NodeLauncher-0000318750: Gathering host keys for node 0000318750 2018-11-29 21:10:42,357 INFO nodepool.NodeLauncher-0000318750: Node id 0000318750 is ready 2018-11-29 21:10:43,270 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1126866340637207s 2018-11-29 21:10:43,282 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:10:43,445 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318751 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:10:43,445 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:10:43,448 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318751, state=deleting, external_id=None 2018-11-29 21:10:44,137 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:44,441 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:10:44,500 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Declining node request 100-0000331945 because node type(s) [runc-centos] not available 2018-11-29 21:10:44,514 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 100-0000331945 2018-11-29 21:10:44,515 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:10:44,872 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.590536117553711s 2018-11-29 21:10:45,569 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.4322266578674316s 2018-11-29 21:10:45,651 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:10:46,377 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Assigning node request 2018-11-29 21:10:46,403 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:10:46,530 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:10:46,620 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.small]: Declining node request 100-0000331945 because node type(s) [runc-centos] not available 2018-11-29 21:10:46,681 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Removing handler for request 100-0000331945 2018-11-29 21:10:46,681 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:10:46,745 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:10:46,870 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:10:46,890 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:10:46,895 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:10:46,903 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:10:46,909 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:10:46,925 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:10:46,926 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:10:46,948 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Fulfilled node request 100-0000331944 2018-11-29 21:10:46,950 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Unlocked node 0000318750 for request 100-0000331944 2018-11-29 21:10:46,956 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 100-0000331944 2018-11-29 21:10:46,956 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:10:46,970 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 100-0000331945 because node type(s) [runc-centos] not available 2018-11-29 21:10:46,986 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 100-0000331945 2018-11-29 21:10:46,986 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:10:48,728 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:10:48,775 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:48,869 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.1404101848602295s 2018-11-29 21:10:48,869 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:10:48,870 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:10:49,068 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1981205940246582s 2018-11-29 21:10:49,072 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:49,674 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6023082733154297s 2018-11-29 21:10:49,693 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:10:49,693 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:10:49,810 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:10:49,863 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:10:49,863 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:10:49,863 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:10:49,874 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318753 for request 300-0000331925 2018-11-29 21:10:49,880 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:10:49,881 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:10:49,882 INFO nodepool.NodeLauncher-0000318753: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318753 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318753 2018-11-29 21:10:49,883 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:10:50,201 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.426245927810669s 2018-11-29 21:10:50,901 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:10:51,546 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.663419246673584s 2018-11-29 21:10:51,547 ERROR nodepool.NodeLauncher-0000318753: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318753: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:10:51,547 INFO nodepool.NodeLauncher-0000318753: Quota exceeded, invalidating quota cache 2018-11-29 21:10:51,547 INFO nodepool.NodeLauncher-0000318753: Aborting node 0000318753 due to quota failure 2018-11-29 21:10:51,585 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:52,431 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.5296673774719238s 2018-11-29 21:10:52,606 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.020280122756958s 2018-11-29 21:10:54,175 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318752 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:10:54,175 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:10:54,178 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318752, state=deleting, external_id=None 2018-11-29 21:10:54,570 DEBUG nodepool.NodePool: No more active min-ready requests for label dib-dci-centos-7 2018-11-29 21:10:54,658 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:10:55,301 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:55,730 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:10:56,467 INFO nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Assigning node request 2018-11-29 21:10:56,521 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-tripleo-ovb]: Declining node request 100-0000331945 because node type(s) [runc-centos] not available 2018-11-29 21:10:56,535 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Removing handler for request 100-0000331945 2018-11-29 21:10:56,535 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:10:56,604 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 21:10:56,657 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 100-0000331945 because node type(s) [runc-centos] not available 2018-11-29 21:10:56,673 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 100-0000331945 2018-11-29 21:10:56,674 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:10:56,736 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.435333490371704s 2018-11-29 21:10:56,755 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:10:57,165 INFO nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Assigning node request 2018-11-29 21:10:57,251 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:10:57,260 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:10:57,272 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:10:57,285 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:10:57,295 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:10:57,300 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:10:57,311 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:10:57,316 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:10:57,341 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4]: Declining node request 100-0000331945 because node type(s) [runc-centos] not available 2018-11-29 21:10:57,357 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Removing handler for request 100-0000331945 2018-11-29 21:10:57,357 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:10:58,323 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:58,471 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:10:59,007 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6842105388641357s 2018-11-29 21:10:59,875 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:10:59,885 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:10:59,985 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.5141839981079102s 2018-11-29 21:11:00,031 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.14534854888916016s 2018-11-29 21:11:00,031 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:11:00,032 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:11:00,234 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.20191407203674316s 2018-11-29 21:11:00,237 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:00,420 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.5452535152435303s 2018-11-29 21:11:00,421 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-cc2eff43-e968-4c17-905f-126122cfd503) 2018-11-29 21:11:00,557 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:11:00,862 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.625117301940918s 2018-11-29 21:11:00,882 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:11:00,882 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:11:00,996 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:11:01,056 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:11:01,056 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:11:01,056 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:11:01,066 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318754 for request 300-0000331819 2018-11-29 21:11:01,073 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:11:01,074 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:11:01,075 INFO nodepool.NodeLauncher-0000318754: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318754 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318754 2018-11-29 21:11:01,075 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:11:02,005 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:02,283 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2072343826293945s 2018-11-29 21:11:02,283 ERROR nodepool.NodeLauncher-0000318754: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318754: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:11:02,283 INFO nodepool.NodeLauncher-0000318754: Quota exceeded, invalidating quota cache 2018-11-29 21:11:02,284 INFO nodepool.NodeLauncher-0000318754: Aborting node 0000318754 due to quota failure 2018-11-29 21:11:02,940 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9348442554473877s 2018-11-29 21:11:04,593 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318634 (state: used, allocated_to: 200-0000331942) 2018-11-29 21:11:04,593 INFO nodepool.DeletedNodeWorker: Deleting used instance e3bca44f-6006-4a1e-8883-76b797b21e91 from rdo-cloud 2018-11-29 21:11:04,596 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:04,911 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318753 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:11:04,911 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:11:04,918 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318753, state=deleting, external_id=None 2018-11-29 21:11:05,019 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:11:05,020 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Declining node request 200-0000331946 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:11:05,051 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 200-0000331946 2018-11-29 21:11:05,052 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:11:05,054 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:11:05,447 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8507936000823975s 2018-11-29 21:11:05,453 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server e3bca44f-6006-4a1e-8883-76b797b21e91 2018-11-29 21:11:05,457 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:11:05,802 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Assigning node request 2018-11-29 21:11:05,803 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.small]: Declining node request 200-0000331946 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:11:05,816 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Removing handler for request 200-0000331946 2018-11-29 21:11:05,816 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:11:05,851 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.39433956146240234s 2018-11-29 21:11:05,879 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkDeleteFloatingips (queue 0) 2018-11-29 21:11:06,231 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1760156154632568s 2018-11-29 21:11:06,600 INFO nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Assigning node request 2018-11-29 21:11:06,600 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-tripleo-ovb]: Declining node request 200-0000331946 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:11:06,612 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Removing handler for request 200-0000331946 2018-11-29 21:11:06,612 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:11:06,743 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 21:11:06,744 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 200-0000331946 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:11:06,755 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 200-0000331946 2018-11-29 21:11:06,755 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:11:06,828 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Assigning node request 2018-11-29 21:11:06,829 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.small]: Declining node request 200-0000331946 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:11:06,846 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Removing handler for request 200-0000331946 2018-11-29 21:11:06,846 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:11:07,148 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkDeleteFloatingips in 1.2682909965515137s 2018-11-29 21:11:07,506 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Assigning node request 2018-11-29 21:11:07,589 INFO nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Assigning node request 2018-11-29 21:11:07,591 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-rdo-ca-ymq-1-main]: Declining node request 200-0000331946 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:11:07,658 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Removing handler for request 200-0000331946 2018-11-29 21:11:07,658 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:11:07,659 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:11:07,695 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Assigning node request 2018-11-29 21:11:07,696 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.large]: Declining node request 200-0000331946 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:11:07,716 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:11:07,739 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:11:07,740 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:11:07,751 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:11:07,757 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Removing handler for request 200-0000331946 2018-11-29 21:11:07,757 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:11:07,768 INFO nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Assigning node request 2018-11-29 21:11:07,769 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4]: Declining node request 200-0000331946 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:11:07,785 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Removing handler for request 200-0000331946 2018-11-29 21:11:07,785 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:11:07,795 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.large]: Declining node request 100-0000331945 because node type(s) [runc-centos] not available 2018-11-29 21:11:07,815 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Assigning node request 2018-11-29 21:11:07,816 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.large]: Declining node request 200-0000331946 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:11:07,829 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Removing handler for request 100-0000331945 2018-11-29 21:11:07,829 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Removing handler for request 200-0000331946 2018-11-29 21:11:07,829 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:11:08,079 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:09,284 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2053797245025635s 2018-11-29 21:11:10,073 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318621 (state: used, allocated_to: 200-0000331938) 2018-11-29 21:11:10,073 INFO nodepool.DeletedNodeWorker: Deleting used instance 0000318621-runc-centos-100-0000331914 from managed-runc-provider 2018-11-29 21:11:10,076 DEBUG nodepool.driver.runc.RuncProvider: Running 'ansible-playbook /opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/runc/playbooks/delete.yml -i /var/opt/rh/rh-python35/lib/nodepool/runc/oci01.softwarefactory-project.io.inventory -e use_rootfs=True -e hypervisor_info_file=/var/opt/rh/rh-python35/lib/nodepool/runc/oci01.softwarefactory-project.io.json -e container_id=0000318621-runc-centos-100-0000331914' 2018-11-29 21:11:10,640 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:11,069 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.4282844066619873s 2018-11-29 21:11:11,081 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:11:11,283 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.20155811309814453s 2018-11-29 21:11:11,284 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:11:11,284 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:11:11,410 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.12599754333496094s 2018-11-29 21:11:11,413 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:11,910 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.4959843158721924s 2018-11-29 21:11:11,928 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:11:11,928 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:11:12,064 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:11:12,114 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:11:12,114 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:11:12,114 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:11:12,124 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318755 for request 300-0000331925 2018-11-29 21:11:12,138 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:11:12,138 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:11:12,139 INFO nodepool.NodeLauncher-0000318755: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318755 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318755 2018-11-29 21:11:12,140 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:11:12,153 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:11:12,232 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.07892251014709473s 2018-11-29 21:11:12,258 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeDeleteServers (queue 0) 2018-11-29 21:11:12,264 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:11:12,497 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeDeleteServers in 0.23854660987854004s 2018-11-29 21:11:12,499 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetPorts (queue 0) 2018-11-29 21:11:12,705 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetPorts in 0.20639324188232422s 2018-11-29 21:11:12,933 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318621, state=deleting, external_id=0000318621-runc-centos-100-0000331914 2018-11-29 21:11:13,169 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.0286898612976074s 2018-11-29 21:11:13,169 ERROR nodepool.NodeLauncher-0000318755: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318755: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:11:13,170 INFO nodepool.NodeLauncher-0000318755: Quota exceeded, invalidating quota cache 2018-11-29 21:11:13,170 INFO nodepool.NodeLauncher-0000318755: Aborting node 0000318755 due to quota failure 2018-11-29 21:11:13,581 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.317087173461914s 2018-11-29 21:11:14,777 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:15,221 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:11:15,587 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318754 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:11:15,587 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:11:15,589 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318754, state=deleting, external_id=None 2018-11-29 21:11:15,832 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0538434982299805s 2018-11-29 21:11:15,896 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:11:16,686 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:11:16,715 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:16,823 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:11:16,917 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:11:17,601 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8861846923828125s 2018-11-29 21:11:17,607 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318634, state=deleting, external_id=e3bca44f-6006-4a1e-8883-76b797b21e91 2018-11-29 21:11:17,781 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:11:17,790 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:11:18,051 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Assigning node request 2018-11-29 21:11:18,052 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1]: Declining node request 200-0000331946 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:11:18,112 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Removing handler for request 200-0000331946 2018-11-29 21:11:18,112 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:11:18,132 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Declining node request 100-0000331945 because node type(s) [runc-centos] not available 2018-11-29 21:11:18,134 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:11:18,150 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Accepting node request 200-0000331946 2018-11-29 21:11:18,166 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:11:18,167 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:11:18,171 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:11:18,179 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:11:18,189 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 100-0000331945 2018-11-29 21:11:18,189 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:11:18,198 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:11:18,255 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Locked existing node 0000318750 for request 200-0000331946 2018-11-29 21:11:18,259 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Fulfilled node request 200-0000331946 2018-11-29 21:11:18,261 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Unlocked node 0000318750 for request 200-0000331946 2018-11-29 21:11:18,268 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 200-0000331946 2018-11-29 21:11:18,268 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:11:19,616 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:11:20,901 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.284942626953125s 2018-11-29 21:11:20,918 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:21,692 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.7712082862854004s 2018-11-29 21:11:22,143 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:11:22,371 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.22839760780334473s 2018-11-29 21:11:22,372 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:11:22,372 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:11:22,622 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.24956107139587402s 2018-11-29 21:11:22,625 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:23,070 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:23,098 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.4726886749267578s 2018-11-29 21:11:23,115 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:11:23,116 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:11:23,214 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:11:23,261 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:11:23,261 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:11:23,261 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:11:23,272 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318756 for request 300-0000331819 2018-11-29 21:11:23,279 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:11:23,279 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:11:23,281 INFO nodepool.NodeLauncher-0000318756: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318756 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318756 2018-11-29 21:11:23,282 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:11:23,964 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8941450119018555s 2018-11-29 21:11:24,650 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.3682637214660645s 2018-11-29 21:11:24,650 ERROR nodepool.NodeLauncher-0000318756: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318756: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:11:24,651 INFO nodepool.NodeLauncher-0000318756: Quota exceeded, invalidating quota cache 2018-11-29 21:11:24,651 INFO nodepool.NodeLauncher-0000318756: Aborting node 0000318756 due to quota failure 2018-11-29 21:11:25,423 INFO nodepool.NodePool: Creating requests for 1 dib-dci-centos-7 nodes 2018-11-29 21:11:25,427 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:11:25,977 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Assigning node request 2018-11-29 21:11:26,025 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.small]: Declining node request 100-0000331947 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:11:26,037 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Removing handler for request 100-0000331947 2018-11-29 21:11:26,037 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:11:26,781 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318755 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:11:26,781 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:11:26,787 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318755, state=deleting, external_id=None 2018-11-29 21:11:26,899 INFO nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Assigning node request 2018-11-29 21:11:26,931 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:27,017 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:11:27,031 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-tripleo-ovb]: Declining node request 100-0000331947 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:11:27,050 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Removing handler for request 100-0000331947 2018-11-29 21:11:27,050 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:11:27,060 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Assigning node request 2018-11-29 21:11:27,118 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.small]: Declining node request 100-0000331947 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:11:27,132 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Removing handler for request 100-0000331947 2018-11-29 21:11:27,132 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:11:27,868 INFO nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Assigning node request 2018-11-29 21:11:27,919 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-rdo-ca-ymq-1-main]: Declining node request 100-0000331945 because node type(s) [runc-centos] not available 2018-11-29 21:11:27,934 INFO nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Assigning node request 2018-11-29 21:11:27,938 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.007411003112793s 2018-11-29 21:11:27,952 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:11:28,004 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-rdo-ca-ymq-1-main]: Declining node request 100-0000331947 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:11:28,016 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Removing handler for request 100-0000331945 2018-11-29 21:11:28,016 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Removing handler for request 100-0000331947 2018-11-29 21:11:28,017 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:11:28,245 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Assigning node request 2018-11-29 21:11:28,510 INFO nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Assigning node request 2018-11-29 21:11:28,529 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:11:28,545 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:11:28,572 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:11:28,580 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:11:28,586 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:11:28,586 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1]: Declining node request 100-0000331947 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:11:28,606 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Removing handler for request 100-0000331947 2018-11-29 21:11:28,606 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:11:28,614 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:11:28,653 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Accepting node request 100-0000331945 2018-11-29 21:11:28,689 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Accepting node request 100-0000331947 2018-11-29 21:11:28,740 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Locked building node 0000318757 for request 100-0000331945 2018-11-29 21:11:28,743 DEBUG nodepool.NodeLauncher-0000318757: Starting runc-centos container 2018-11-29 21:11:28,744 DEBUG nodepool.driver.runc.RuncProvider: Running 'ansible-playbook /opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/runc/playbooks/create.yml -i /var/opt/rh/rh-python35/lib/nodepool/runc/oci01.softwarefactory-project.io.inventory -e use_rootfs=True -e hypervisor_info_file=/var/opt/rh/rh-python35/lib/nodepool/runc/oci01.softwarefactory-project.io.json -e container_id=0000318757-runc-centos-100-0000331945 -e container_spec=/var/opt/rh/rh-python35/lib/nodepool/runc/0000318757-runc-centos-100-0000331945.config -e worker_username=zuul-worker -e worker_homedir=/home/zuul-worker -e host_addr=38.145.33.82 -e container_port=38170' 2018-11-29 21:11:28,755 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: ['100-0000331945'] 2018-11-29 21:11:28,812 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:11:28,829 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:11:28,837 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Predicted remaining provider quota: {'compute': {'cores': 415, 'ram': 374784, 'instances': 45}} 2018-11-29 21:11:28,896 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 14}} 2018-11-29 21:11:28,896 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 13}} 2018-11-29 21:11:28,907 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Locked building node 0000318758 for request 100-0000331947 2018-11-29 21:11:28,912 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: ['100-0000331947'] 2018-11-29 21:11:28,914 INFO nodepool.NodeLauncher-0000318758: Creating server with hostname dib-dci-centos-7-rdo-cloud-0000318758 in rdo-cloud from image dib-dci-centos-7 for node id: 0000318758 2018-11-29 21:11:28,915 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputePostServers (queue 0) 2018-11-29 21:11:29,295 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3424160480499268s 2018-11-29 21:11:30,005 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputePostServers in 1.0895907878875732s 2018-11-29 21:11:30,005 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:31,127 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.1215846538543701s 2018-11-29 21:11:31,135 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServers (queue 0) 2018-11-29 21:11:31,459 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServers in 0.3243277072906494s 2018-11-29 21:11:31,464 DEBUG nodepool.NodeLauncher-0000318758: Waiting for server d6c7fb39-8f4d-47ac-9736-b8a154728484 for node id: 0000318758 2018-11-29 21:11:32,080 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318552 (state: used, allocated_to: 300-0000331939) 2018-11-29 21:11:32,080 INFO nodepool.DeletedNodeWorker: Deleting used instance 6f0abd73-9ab5-46a8-8484-2b304b727409 from rdo-cloud 2018-11-29 21:11:32,086 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 6f0abd73-9ab5-46a8-8484-2b304b727409 2018-11-29 21:11:32,089 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:11:32,266 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.17692232131958008s 2018-11-29 21:11:32,295 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkDeleteFloatingips (queue 0) 2018-11-29 21:11:33,017 INFO nodepool.NodeLauncher-0000318757: Container id 0000318757 is ready 2018-11-29 21:11:33,111 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:33,283 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkDeleteFloatingips in 0.9878153800964355s 2018-11-29 21:11:33,286 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:11:33,481 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.19482874870300293s 2018-11-29 21:11:33,481 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:11:33,482 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:11:33,698 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.21589994430541992s 2018-11-29 21:11:33,701 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:33,920 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.809006929397583s 2018-11-29 21:11:34,289 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5872492790222168s 2018-11-29 21:11:34,309 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:11:34,310 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:11:34,437 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:11:34,496 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:11:34,496 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:11:34,496 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:11:34,507 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318759 for request 300-0000331925 2018-11-29 21:11:34,515 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:11:34,516 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:11:34,517 INFO nodepool.NodeLauncher-0000318759: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318759 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318759 2018-11-29 21:11:34,518 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:11:35,326 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:11:35,637 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:11:35,701 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.1834080219268799s 2018-11-29 21:11:35,702 ERROR nodepool.NodeLauncher-0000318759: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318759: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:11:35,702 INFO nodepool.NodeLauncher-0000318759: Quota exceeded, invalidating quota cache 2018-11-29 21:11:35,702 INFO nodepool.NodeLauncher-0000318759: Aborting node 0000318759 due to quota failure 2018-11-29 21:11:36,104 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:11:36,469 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:36,707 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3809096813201904s 2018-11-29 21:11:37,097 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:11:37,114 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6447114944458008s 2018-11-29 21:11:37,122 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetPorts (queue 0) 2018-11-29 21:11:37,138 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:11:37,208 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:11:37,329 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetPorts in 0.2068464756011963s 2018-11-29 21:11:37,334 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:11:37,521 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.18644976615905762s 2018-11-29 21:11:37,549 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkPostFloatingips (queue 0) 2018-11-29 21:11:37,569 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318756 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:11:37,569 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:11:37,573 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318756, state=deleting, external_id=None 2018-11-29 21:11:38,075 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:11:38,713 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:11:38,760 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:11:38,833 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:11:38,841 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:11:38,844 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:11:38,851 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:11:38,893 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Fulfilled node request 100-0000331945 2018-11-29 21:11:38,897 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Unlocked node 0000318757 for request 100-0000331945 2018-11-29 21:11:38,902 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Removing handler for request 100-0000331945 2018-11-29 21:11:38,902 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:11:38,979 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: ['100-0000331947'] 2018-11-29 21:11:39,182 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:40,163 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9809901714324951s 2018-11-29 21:11:40,686 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkPostFloatingips in 3.137181520462036s 2018-11-29 21:11:40,690 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeDeleteServers (queue 0) 2018-11-29 21:11:40,920 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeDeleteServers in 0.2300245761871338s 2018-11-29 21:11:42,735 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:11:42,925 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:44,011 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.0855717658996582s 2018-11-29 21:11:44,019 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetPorts (queue 0) 2018-11-29 21:11:44,235 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetPorts in 0.21634173393249512s 2018-11-29 21:11:44,421 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.6861016750335693s 2018-11-29 21:11:44,522 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:11:44,720 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.1983656883239746s 2018-11-29 21:11:44,721 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:11:44,722 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:11:44,852 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1300053596496582s 2018-11-29 21:11:44,855 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:45,243 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:45,405 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5496311187744141s 2018-11-29 21:11:45,423 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:11:45,423 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:11:45,544 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:11:45,614 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:11:45,614 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:11:45,614 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:11:45,627 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318760 for request 300-0000331819 2018-11-29 21:11:45,633 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:11:45,634 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:11:45,635 INFO nodepool.NodeLauncher-0000318760: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318760 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318760 2018-11-29 21:11:45,635 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:11:45,703 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:11:45,788 DEBUG nodepool.NodePool: No more active min-ready requests for label runc-centos 2018-11-29 21:11:45,803 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.0993490219116211s 2018-11-29 21:11:45,835 DEBUG nodepool.NodeLauncher-0000318758: Node 0000318758 is running [region: regionOne, az: nova, ip: 38.145.33.20 ipv4: 38.145.33.20, ipv6: ] 2018-11-29 21:11:45,836 DEBUG nodepool.NodeLauncher-0000318758: Gathering host keys for node 0000318758 2018-11-29 21:11:45,865 INFO nodepool.NodeLauncher-0000318758: Node id 0000318758 is ready 2018-11-29 21:11:45,934 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:11:46,183 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:11:46,388 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1442012786865234s 2018-11-29 21:11:47,179 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:11:47,223 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:11:47,280 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.6444931030273438s 2018-11-29 21:11:47,280 ERROR nodepool.NodeLauncher-0000318760: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318760: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:11:47,281 INFO nodepool.NodeLauncher-0000318760: Quota exceeded, invalidating quota cache 2018-11-29 21:11:47,281 INFO nodepool.NodeLauncher-0000318760: Aborting node 0000318760 due to quota failure 2018-11-29 21:11:47,295 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:11:48,109 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318759 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:11:48,110 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:11:48,114 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318759, state=deleting, external_id=None 2018-11-29 21:11:48,261 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:11:48,791 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:11:48,839 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:11:49,057 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:11:49,078 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:11:49,079 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:11:49,085 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:11:49,105 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:11:49,124 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Fulfilled node request 100-0000331947 2018-11-29 21:11:49,127 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Unlocked node 0000318758 for request 100-0000331947 2018-11-29 21:11:49,134 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 100-0000331947 2018-11-29 21:11:49,134 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:11:49,163 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:49,939 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7762460708618164s 2018-11-29 21:11:50,245 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318552, state=deleting, external_id=6f0abd73-9ab5-46a8-8484-2b304b727409 2018-11-29 21:11:50,451 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:11:51,838 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.38716459274292s 2018-11-29 21:11:51,838 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:52,933 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.094261646270752s 2018-11-29 21:11:55,639 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:11:55,858 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.21889400482177734s 2018-11-29 21:11:55,859 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:11:55,859 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:11:55,963 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:56,064 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.20516037940979004s 2018-11-29 21:11:56,069 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:56,086 DEBUG nodepool.NodePool: No more active min-ready requests for label dib-dci-centos-7 2018-11-29 21:11:56,172 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:11:56,173 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Accepting node request 100-0000331948 2018-11-29 21:11:56,274 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:11:56,331 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Locked existing node 0000318745 for request 100-0000331948 2018-11-29 21:11:56,335 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Fulfilled node request 100-0000331948 2018-11-29 21:11:56,338 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Unlocked node 0000318745 for request 100-0000331948 2018-11-29 21:11:56,344 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 100-0000331948 2018-11-29 21:11:56,344 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:11:56,400 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.4370408058166504s 2018-11-29 21:11:56,607 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5371696949005127s 2018-11-29 21:11:56,697 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:11:56,697 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:11:56,830 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:11:56,889 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:11:56,889 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:11:56,889 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:11:56,899 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318761 for request 300-0000331925 2018-11-29 21:11:56,907 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:11:56,907 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:11:56,909 INFO nodepool.NodeLauncher-0000318761: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318761 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318761 2018-11-29 21:11:56,909 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:11:57,250 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:11:57,301 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:11:57,371 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:11:57,860 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.950660228729248s 2018-11-29 21:11:57,860 ERROR nodepool.NodeLauncher-0000318761: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318761: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:11:57,861 INFO nodepool.NodeLauncher-0000318761: Quota exceeded, invalidating quota cache 2018-11-29 21:11:57,861 INFO nodepool.NodeLauncher-0000318761: Aborting node 0000318761 due to quota failure 2018-11-29 21:11:57,866 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:11:58,333 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:11:58,750 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318760 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:11:58,750 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:11:58,754 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318760, state=deleting, external_id=None 2018-11-29 21:11:58,836 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318711 (state: used, allocated_to: 100-0000331936) 2018-11-29 21:11:58,836 INFO nodepool.DeletedNodeWorker: Deleting used instance None from ansible-statics 2018-11-29 21:11:58,840 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318711, state=deleting, external_id=None 2018-11-29 21:11:58,975 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:11:58,997 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:11:59,155 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2889142036437988s 2018-11-29 21:11:59,161 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:11:59,309 DEBUG nodepool.driver.static.StaticNodeProvider: Registered static node 35.230.187.160 2018-11-29 21:11:59,317 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:11:59,348 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:11:59,349 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:11:59,355 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:11:59,369 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:11:59,378 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:12:00,680 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.518399953842163s 2018-11-29 21:12:00,967 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:01,377 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.4097440242767334s 2018-11-29 21:12:01,377 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-617b8a98-2c7d-47af-aff8-d33319dbd113) 2018-11-29 21:12:01,494 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:12:02,037 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:03,330 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.2919795513153076s 2018-11-29 21:12:04,075 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318741 (state: used, allocated_to: 300-0000331940) 2018-11-29 21:12:04,075 INFO nodepool.DeletedNodeWorker: Deleting used instance 5960ec0b-7a59-4481-8d7e-942c1336baf7 from rdo-cloud 2018-11-29 21:12:04,080 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 5960ec0b-7a59-4481-8d7e-942c1336baf7 2018-11-29 21:12:04,083 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:12:04,296 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.21261882781982422s 2018-11-29 21:12:04,323 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkDeleteFloatingips (queue 0) 2018-11-29 21:12:05,189 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:12:05,630 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkDeleteFloatingips in 1.3067066669464111s 2018-11-29 21:12:06,397 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:12:06,447 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:12:06,559 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3706872463226318s 2018-11-29 21:12:06,575 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:06,912 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:12:07,124 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.21092939376831055s 2018-11-29 21:12:07,124 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:12:07,124 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:12:07,252 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1274576187133789s 2018-11-29 21:12:07,255 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:07,339 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:12:07,384 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:12:07,445 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:12:07,685 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.42921948432922363s 2018-11-29 21:12:07,704 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:12:07,705 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:12:07,806 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2291374206542969s 2018-11-29 21:12:07,848 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:12:07,911 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:12:07,911 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:12:07,911 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:12:07,921 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318763 for request 300-0000331819 2018-11-29 21:12:07,928 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:12:07,928 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:12:07,932 INFO nodepool.NodeLauncher-0000318763: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318763 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318763 2018-11-29 21:12:07,932 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:12:08,412 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:12:08,425 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:09,058 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:12:09,080 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:12:09,146 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.720435380935669s 2018-11-29 21:12:09,319 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318761 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:12:09,320 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:12:09,324 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318761, state=deleting, external_id=None 2018-11-29 21:12:09,366 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4339599609375s 2018-11-29 21:12:09,367 ERROR nodepool.NodeLauncher-0000318763: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318763: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:12:09,367 INFO nodepool.NodeLauncher-0000318763: Quota exceeded, invalidating quota cache 2018-11-29 21:12:09,367 INFO nodepool.NodeLauncher-0000318763: Aborting node 0000318763 due to quota failure 2018-11-29 21:12:09,480 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318715 (state: used, allocated_to: 100-0000331937) 2018-11-29 21:12:09,480 INFO nodepool.DeletedNodeWorker: Deleting used instance None from ansible-statics 2018-11-29 21:12:09,487 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318715, state=deleting, external_id=None 2018-11-29 21:12:09,738 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:12:09,774 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:12:09,779 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:12:09,780 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:12:09,784 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:12:09,786 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:12:10,106 DEBUG nodepool.driver.static.StaticNodeProvider: Registered static node 35.199.20.133 2018-11-29 21:12:10,637 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:12:10,708 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.07107853889465332s 2018-11-29 21:12:10,737 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeDeleteServers (queue 0) 2018-11-29 21:12:11,147 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeDeleteServers in 0.4107346534729004s 2018-11-29 21:12:11,150 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetPorts (queue 0) 2018-11-29 21:12:11,360 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetPorts in 0.2098989486694336s 2018-11-29 21:12:12,590 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:12:14,096 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.5062365531921387s 2018-11-29 21:12:14,096 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:15,162 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:15,596 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.4991254806518555s 2018-11-29 21:12:15,801 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6390290260314941s 2018-11-29 21:12:16,642 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:12:16,645 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:12:17,372 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318741, state=deleting, external_id=5960ec0b-7a59-4481-8d7e-942c1336baf7 2018-11-29 21:12:17,457 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:12:17,512 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:12:17,544 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:12:17,932 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:12:18,092 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.1593952178955078s 2018-11-29 21:12:18,092 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:12:18,093 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:12:18,219 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.12673234939575195s 2018-11-29 21:12:18,223 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:18,490 INFO nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Assigning node request 2018-11-29 21:12:18,491 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-rdo-ca-ymq-1-main]: Declining node request 200-0000331949 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:18,506 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Removing handler for request 200-0000331949 2018-11-29 21:12:18,506 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:12:18,925 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.7019193172454834s 2018-11-29 21:12:18,943 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:12:18,943 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:12:19,052 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:12:19,143 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:12:19,143 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:12:19,144 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:12:19,156 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318765 for request 300-0000331925 2018-11-29 21:12:19,165 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:12:19,165 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:12:19,169 INFO nodepool.NodeLauncher-0000318765: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318765 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318765 2018-11-29 21:12:19,170 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:12:19,182 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:12:19,182 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 200-0000331949 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:19,197 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 200-0000331949 2018-11-29 21:12:19,198 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:12:19,210 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Assigning node request 2018-11-29 21:12:19,210 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.large]: Declining node request 200-0000331949 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:19,225 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Removing handler for request 200-0000331949 2018-11-29 21:12:19,225 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:12:20,011 INFO nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Assigning node request 2018-11-29 21:12:20,012 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4]: Declining node request 200-0000331949 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:20,084 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Removing handler for request 200-0000331949 2018-11-29 21:12:20,084 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:12:20,125 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:12:20,163 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Assigning node request 2018-11-29 21:12:20,164 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.large]: Declining node request 200-0000331949 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:20,178 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:12:20,185 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:12:20,188 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:12:20,189 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:12:20,212 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Removing handler for request 200-0000331949 2018-11-29 21:12:20,212 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:12:20,251 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318763 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:12:20,252 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:12:20,256 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318763, state=deleting, external_id=None 2018-11-29 21:12:20,293 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318750 (state: used, allocated_to: 200-0000331946) 2018-11-29 21:12:20,293 INFO nodepool.DeletedNodeWorker: Deleting used instance 82795ae5-7aa5-4ab7-ac8a-6e5adde268d3 from rdo-cloud 2018-11-29 21:12:20,297 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 82795ae5-7aa5-4ab7-ac8a-6e5adde268d3 2018-11-29 21:12:20,301 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:12:20,351 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.1811349391937256s 2018-11-29 21:12:20,352 ERROR nodepool.NodeLauncher-0000318765: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318765: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:12:20,352 INFO nodepool.NodeLauncher-0000318765: Quota exceeded, invalidating quota cache 2018-11-29 21:12:20,352 INFO nodepool.NodeLauncher-0000318765: Aborting node 0000318765 due to quota failure 2018-11-29 21:12:20,570 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.26856064796447754s 2018-11-29 21:12:20,595 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkDeleteFloatingips (queue 0) 2018-11-29 21:12:21,478 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3523821830749512s 2018-11-29 21:12:21,478 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:21,568 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkDeleteFloatingips in 0.9734539985656738s 2018-11-29 21:12:21,569 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:22,151 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.5821952819824219s 2018-11-29 21:12:22,742 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.263688325881958s 2018-11-29 21:12:26,582 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:12:26,723 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.14086055755615234s 2018-11-29 21:12:26,805 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeDeleteServers (queue 0) 2018-11-29 21:12:26,956 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Assigning node request 2018-11-29 21:12:26,958 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.small]: Declining node request 200-0000331949 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:26,960 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:12:26,972 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Removing handler for request 200-0000331949 2018-11-29 21:12:26,972 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:12:27,386 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeDeleteServers in 0.5751347541809082s 2018-11-29 21:12:27,387 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:27,510 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:12:27,563 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 21:12:27,563 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 200-0000331949 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:27,587 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 200-0000331949 2018-11-29 21:12:27,587 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:12:27,634 INFO nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Assigning node request 2018-11-29 21:12:27,634 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-tripleo-ovb]: Declining node request 200-0000331949 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:27,653 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Removing handler for request 200-0000331949 2018-11-29 21:12:27,653 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:12:27,670 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Assigning node request 2018-11-29 21:12:27,670 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.small]: Declining node request 200-0000331949 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:27,686 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Removing handler for request 200-0000331949 2018-11-29 21:12:27,686 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:12:28,189 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8019084930419922s 2018-11-29 21:12:28,196 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetPorts (queue 0) 2018-11-29 21:12:28,422 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetPorts in 0.2256779670715332s 2018-11-29 21:12:28,579 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:12:28,712 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2024469375610352s 2018-11-29 21:12:28,728 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:29,170 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:12:29,279 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:12:29,301 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.13126134872436523s 2018-11-29 21:12:29,302 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:12:29,302 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:12:29,313 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:12:29,560 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.25751781463623047s 2018-11-29 21:12:29,563 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:29,924 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.36060214042663574s 2018-11-29 21:12:29,942 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:12:29,942 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:12:30,078 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:12:30,179 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:12:30,179 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:12:30,179 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:12:30,194 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318766 for request 300-0000331819 2018-11-29 21:12:30,205 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:12:30,206 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:12:30,213 INFO nodepool.NodeLauncher-0000318766: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318766 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318766 2018-11-29 21:12:30,214 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:12:30,243 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:12:30,444 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:12:30,446 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Declining node request 200-0000331949 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:30,451 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:12:30,452 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:12:30,455 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:12:30,455 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:12:30,460 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.7281749248504639s 2018-11-29 21:12:30,484 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 200-0000331949 2018-11-29 21:12:30,485 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:12:30,791 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318765 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:12:30,791 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:12:30,796 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318765, state=deleting, external_id=None 2018-11-29 21:12:31,748 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.5343191623687744s 2018-11-29 21:12:31,749 ERROR nodepool.NodeLauncher-0000318766: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318766: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:12:31,749 INFO nodepool.NodeLauncher-0000318766: Quota exceeded, invalidating quota cache 2018-11-29 21:12:31,749 INFO nodepool.NodeLauncher-0000318766: Aborting node 0000318766 due to quota failure 2018-11-29 21:12:32,434 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:12:32,496 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.06235671043395996s 2018-11-29 21:12:33,836 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:34,244 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.40752434730529785s 2018-11-29 21:12:34,521 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318750, state=deleting, external_id=82795ae5-7aa5-4ab7-ac8a-6e5adde268d3 2018-11-29 21:12:34,741 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:12:35,894 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1528594493865967s 2018-11-29 21:12:35,900 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:37,145 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2450828552246094s 2018-11-29 21:12:37,225 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:12:37,225 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Declining node request 200-0000331949 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:37,233 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:12:37,245 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 200-0000331949 2018-11-29 21:12:37,245 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:12:37,670 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:12:37,739 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:12:37,770 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:12:38,646 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:12:39,349 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:12:39,390 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:12:40,185 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:40,210 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:12:40,349 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:12:40,423 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.21309947967529297s 2018-11-29 21:12:40,423 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:12:40,426 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:12:40,633 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.20731139183044434s 2018-11-29 21:12:40,637 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:40,664 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:12:40,673 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:12:40,679 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:12:40,683 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Accepting node request 200-0000331949 2018-11-29 21:12:40,683 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:12:40,698 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:12:40,754 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Locked existing node 0000318758 for request 200-0000331949 2018-11-29 21:12:40,759 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Fulfilled node request 200-0000331949 2018-11-29 21:12:40,761 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Unlocked node 0000318758 for request 200-0000331949 2018-11-29 21:12:40,769 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 200-0000331949 2018-11-29 21:12:40,769 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:12:40,871 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6855688095092773s 2018-11-29 21:12:41,260 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.623053789138794s 2018-11-29 21:12:41,279 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:12:41,279 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:12:41,443 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:12:41,510 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:12:41,510 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:12:41,510 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:12:41,522 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318767 for request 300-0000331925 2018-11-29 21:12:41,530 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:12:41,531 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:12:41,532 INFO nodepool.NodeLauncher-0000318767: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318767 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318767 2018-11-29 21:12:41,533 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:12:41,926 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:12:42,986 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.453049898147583s 2018-11-29 21:12:42,986 ERROR nodepool.NodeLauncher-0000318767: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318767: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:12:42,987 INFO nodepool.NodeLauncher-0000318767: Quota exceeded, invalidating quota cache 2018-11-29 21:12:42,987 INFO nodepool.NodeLauncher-0000318767: Aborting node 0000318767 due to quota failure 2018-11-29 21:12:43,189 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.26285719871521s 2018-11-29 21:12:43,204 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:44,178 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9718255996704102s 2018-11-29 21:12:46,270 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:46,521 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318766 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:12:46,521 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:12:46,524 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318766, state=deleting, external_id=None 2018-11-29 21:12:46,691 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.42034292221069336s 2018-11-29 21:12:47,536 INFO nodepool.NodePool: Creating requests for 1 dib-dci-centos-7 nodes 2018-11-29 21:12:47,540 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:12:47,541 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:12:47,770 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 21:12:47,893 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 100-0000331950 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:47,894 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:12:47,911 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Assigning node request 2018-11-29 21:12:47,912 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 100-0000331950 2018-11-29 21:12:47,912 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:12:47,983 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.small]: Declining node request 100-0000331950 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:48,001 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Removing handler for request 100-0000331950 2018-11-29 21:12:48,001 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:12:48,728 INFO nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Assigning node request 2018-11-29 21:12:48,784 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-rdo-ca-ymq-1-main]: Declining node request 100-0000331950 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:48,799 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Removing handler for request 100-0000331950 2018-11-29 21:12:48,799 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:12:49,214 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:49,441 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:12:49,505 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:12:49,533 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 100-0000331950 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:49,548 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 100-0000331950 2018-11-29 21:12:49,548 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:12:50,147 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9335503578186035s 2018-11-29 21:12:50,161 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:12:50,462 INFO nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Assigning node request 2018-11-29 21:12:50,517 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4]: Declining node request 100-0000331950 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:50,531 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Removing handler for request 100-0000331950 2018-11-29 21:12:50,531 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:12:50,834 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Assigning node request 2018-11-29 21:12:50,882 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:12:50,896 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:12:50,911 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:12:50,950 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:12:50,961 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.large]: Declining node request 100-0000331950 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:50,977 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Removing handler for request 100-0000331950 2018-11-29 21:12:50,977 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:12:51,411 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2494680881500244s 2018-11-29 21:12:51,535 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:12:51,743 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.20710086822509766s 2018-11-29 21:12:51,743 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:12:51,744 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:12:51,946 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.20218205451965332s 2018-11-29 21:12:51,949 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:52,272 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.3223729133605957s 2018-11-29 21:12:52,290 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:12:52,290 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:12:52,418 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:12:52,475 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:12:52,475 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:12:52,476 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:12:52,489 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318768 for request 300-0000331819 2018-11-29 21:12:52,496 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:12:52,496 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:12:52,497 INFO nodepool.NodeLauncher-0000318768: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318768 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318768 2018-11-29 21:12:52,498 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:12:52,706 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:53,435 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7286593914031982s 2018-11-29 21:12:53,733 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2350847721099854s 2018-11-29 21:12:53,733 ERROR nodepool.NodeLauncher-0000318768: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318768: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:12:53,734 INFO nodepool.NodeLauncher-0000318768: Quota exceeded, invalidating quota cache 2018-11-29 21:12:53,734 INFO nodepool.NodeLauncher-0000318768: Aborting node 0000318768 due to quota failure 2018-11-29 21:12:55,189 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:56,963 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.7735965251922607s 2018-11-29 21:12:57,008 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318767 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:12:57,009 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:12:57,012 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318767, state=deleting, external_id=None 2018-11-29 21:12:57,441 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:12:57,790 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Assigning node request 2018-11-29 21:12:57,793 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:12:57,843 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.small]: Declining node request 100-0000331950 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:57,857 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Removing handler for request 100-0000331950 2018-11-29 21:12:57,857 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:12:57,992 INFO nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Assigning node request 2018-11-29 21:12:58,021 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:12:58,084 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-tripleo-ovb]: Declining node request 100-0000331950 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:58,100 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Removing handler for request 100-0000331950 2018-11-29 21:12:58,100 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:12:58,103 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:12:58,714 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2725751399993896s 2018-11-29 21:12:58,857 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:12:58,907 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:12:59,319 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.4118459224700928s 2018-11-29 21:12:59,588 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Assigning node request 2018-11-29 21:12:59,644 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:12:59,660 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.large]: Declining node request 100-0000331950 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:12:59,674 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Removing handler for request 100-0000331950 2018-11-29 21:12:59,674 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:13:00,631 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:13:01,000 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Assigning node request 2018-11-29 21:13:01,045 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:13:01,093 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:13:01,135 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:13:01,145 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:13:01,148 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1]: Declining node request 100-0000331950 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:13:01,161 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Removing handler for request 100-0000331950 2018-11-29 21:13:01,162 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:13:01,904 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:02,019 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:02,266 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.3616769313812256s 2018-11-29 21:13:02,266 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-5c030daa-bb66-4ab5-ad86-7ec70d7825fc) 2018-11-29 21:13:02,441 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:13:02,500 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:13:02,667 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.16640591621398926s 2018-11-29 21:13:02,668 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:13:02,668 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:13:02,798 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.12949323654174805s 2018-11-29 21:13:02,801 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:03,065 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.04524564743042s 2018-11-29 21:13:03,526 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.7246429920196533s 2018-11-29 21:13:03,543 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:13:03,543 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:13:03,649 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:13:03,696 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:13:03,696 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:13:03,696 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:13:03,706 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318769 for request 300-0000331925 2018-11-29 21:13:03,714 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:13:03,715 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:13:03,716 INFO nodepool.NodeLauncher-0000318769: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318769 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318769 2018-11-29 21:13:03,717 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:13:04,740 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:13:05,264 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.547194242477417s 2018-11-29 21:13:05,265 ERROR nodepool.NodeLauncher-0000318769: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318769: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:13:05,265 INFO nodepool.NodeLauncher-0000318769: Quota exceeded, invalidating quota cache 2018-11-29 21:13:05,265 INFO nodepool.NodeLauncher-0000318769: Aborting node 0000318769 due to quota failure 2018-11-29 21:13:05,335 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:06,070 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.329084873199463s 2018-11-29 21:13:06,090 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7551536560058594s 2018-11-29 21:13:07,476 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318768 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:13:07,476 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:13:07,479 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318768, state=deleting, external_id=None 2018-11-29 21:13:08,053 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:13:08,060 INFO nodepool.PoolWorker.ansible-statics-main: Assigning node request 2018-11-29 21:13:08,090 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:08,177 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:13:08,189 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.ansible-statics-main]: Declining node request 100-0000331950 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:13:08,211 DEBUG nodepool.PoolWorker.ansible-statics-main: Removing handler for request 100-0000331950 2018-11-29 21:13:08,212 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:13:08,239 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:13:08,240 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:13:08,916 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:13:09,048 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9577934741973877s 2018-11-29 21:13:09,720 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:13:09,756 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:13:10,741 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:13:11,136 INFO nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Assigning node request 2018-11-29 21:13:11,266 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:13:11,319 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Declining node request 100-0000331950 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:13:11,336 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:13:11,346 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Removing handler for request 100-0000331950 2018-11-29 21:13:11,346 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:13:11,346 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:13:11,354 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:13:11,466 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:12,092 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6254980564117432s 2018-11-29 21:13:13,068 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:13:13,719 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:13:13,860 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.1405775547027588s 2018-11-29 21:13:13,861 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:13:13,861 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:13:14,003 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.14168405532836914s 2018-11-29 21:13:14,006 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:14,465 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3966128826141357s 2018-11-29 21:13:14,480 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:14,643 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6368007659912109s 2018-11-29 21:13:14,660 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:13:14,660 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:13:14,756 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:13:14,804 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:13:14,804 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:13:14,804 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:13:14,813 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318770 for request 300-0000331819 2018-11-29 21:13:14,819 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:13:14,820 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:13:14,821 INFO nodepool.NodeLauncher-0000318770: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318770 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318770 2018-11-29 21:13:14,822 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:13:15,476 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9953286647796631s 2018-11-29 21:13:16,086 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2641878128051758s 2018-11-29 21:13:16,087 ERROR nodepool.NodeLauncher-0000318770: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318770: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:13:16,087 INFO nodepool.NodeLauncher-0000318770: Quota exceeded, invalidating quota cache 2018-11-29 21:13:16,087 INFO nodepool.NodeLauncher-0000318770: Aborting node 0000318770 due to quota failure 2018-11-29 21:13:17,936 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318769 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:13:17,936 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:13:17,941 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318769, state=deleting, external_id=None 2018-11-29 21:13:18,119 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:18,426 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:13:18,434 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Assigning node request 2018-11-29 21:13:18,459 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:13:18,473 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:13:18,476 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:13:18,521 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org-tripleo]: Declining node request 100-0000331950 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:13:18,537 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Removing handler for request 100-0000331950 2018-11-29 21:13:18,537 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:13:18,769 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6492841243743896s 2018-11-29 21:13:18,975 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:13:19,804 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:13:19,837 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:13:20,493 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:20,841 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:13:21,365 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:13:21,526 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.033212661743164s 2018-11-29 21:13:21,541 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:13:21,560 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Declining node request 100-0000331950 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:13:21,577 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:13:21,584 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:13:21,586 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:13:21,586 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:13:21,594 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 100-0000331950 2018-11-29 21:13:21,594 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:13:22,831 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2898499965667725s 2018-11-29 21:13:24,126 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:24,600 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.4737052917480469s 2018-11-29 21:13:24,824 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:13:25,008 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.18395519256591797s 2018-11-29 21:13:25,009 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:13:25,009 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:13:25,148 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1388261318206787s 2018-11-29 21:13:25,152 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:25,706 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5548107624053955s 2018-11-29 21:13:25,725 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:13:25,726 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:13:25,826 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:13:25,880 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:13:25,880 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:13:25,880 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:13:25,891 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318771 for request 300-0000331925 2018-11-29 21:13:25,898 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:13:25,899 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:13:25,900 INFO nodepool.NodeLauncher-0000318771: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318771 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318771 2018-11-29 21:13:25,901 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:13:26,559 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:27,194 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2933316230773926s 2018-11-29 21:13:27,195 ERROR nodepool.NodeLauncher-0000318771: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318771: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:13:27,196 INFO nodepool.NodeLauncher-0000318771: Quota exceeded, invalidating quota cache 2018-11-29 21:13:27,196 INFO nodepool.NodeLauncher-0000318771: Aborting node 0000318771 due to quota failure 2018-11-29 21:13:27,530 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9706544876098633s 2018-11-29 21:13:28,859 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:13:28,873 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:13:28,879 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:13:28,880 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:13:28,880 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:13:28,881 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:13:28,904 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318770 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:13:28,904 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:13:28,907 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318770, state=deleting, external_id=None 2018-11-29 21:13:29,063 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:13:29,886 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:13:29,919 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:13:30,074 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2142441272735596s 2018-11-29 21:13:30,614 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:30,950 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:13:31,594 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9791259765625s 2018-11-29 21:13:31,795 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:13:31,814 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:13:31,815 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:13:31,816 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:13:31,821 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:13:31,881 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Accepting node request 100-0000331950 2018-11-29 21:13:31,976 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:13:31,994 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:13:32,003 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Predicted remaining provider quota: {'compute': {'cores': 431, 'ram': 391168, 'instances': 47}} 2018-11-29 21:13:32,059 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 14}} 2018-11-29 21:13:32,060 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 13}} 2018-11-29 21:13:32,070 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Locked building node 0000318772 for request 100-0000331950 2018-11-29 21:13:32,075 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: ['100-0000331950'] 2018-11-29 21:13:32,076 INFO nodepool.NodeLauncher-0000318772: Creating server with hostname dib-dci-centos-7-rdo-cloud-0000318772 in rdo-cloud from image dib-dci-centos-7 for node id: 0000318772 2018-11-29 21:13:32,076 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputePostServers (queue 0) 2018-11-29 21:13:32,575 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:33,757 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputePostServers in 1.680887222290039s 2018-11-29 21:13:33,758 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServers (queue 0) 2018-11-29 21:13:33,866 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServers in 0.10783267021179199s 2018-11-29 21:13:33,871 DEBUG nodepool.NodeLauncher-0000318772: Waiting for server 33b4e5b4-c8c0-4582-a09b-677324680c77 for node id: 0000318772 2018-11-29 21:13:34,189 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.6137967109680176s 2018-11-29 21:13:35,904 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:13:36,097 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.19283223152160645s 2018-11-29 21:13:36,098 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:13:36,098 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:13:36,103 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:13:36,289 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.18990802764892578s 2018-11-29 21:13:36,292 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:36,632 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.34021687507629395s 2018-11-29 21:13:36,651 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:13:36,651 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:13:36,762 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:13:36,801 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:36,812 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:13:36,812 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:13:36,812 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:13:36,824 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318773 for request 300-0000331819 2018-11-29 21:13:36,831 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:13:36,831 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:13:36,832 INFO nodepool.NodeLauncher-0000318773: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318773 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318773 2018-11-29 21:13:36,833 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:13:37,476 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6747031211853027s 2018-11-29 21:13:37,485 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3818306922912598s 2018-11-29 21:13:38,369 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.535348653793335s 2018-11-29 21:13:38,369 ERROR nodepool.NodeLauncher-0000318773: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318773: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:13:38,369 INFO nodepool.NodeLauncher-0000318773: Quota exceeded, invalidating quota cache 2018-11-29 21:13:38,369 INFO nodepool.NodeLauncher-0000318773: Aborting node 0000318773 due to quota failure 2018-11-29 21:13:39,435 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:13:39,437 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:13:39,438 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:13:39,438 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:13:39,441 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:13:39,442 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:13:39,505 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:39,558 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318771 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:13:39,558 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:13:39,566 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318771, state=deleting, external_id=None 2018-11-29 21:13:39,969 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:13:40,013 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:13:40,432 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9275491237640381s 2018-11-29 21:13:41,065 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:13:41,942 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:13:41,975 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:13:41,984 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:13:41,986 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:13:42,150 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: ['100-0000331950'] 2018-11-29 21:13:43,493 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:43,906 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.412489652633667s 2018-11-29 21:13:44,453 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:13:44,827 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318758 (state: used, allocated_to: 200-0000331949) 2018-11-29 21:13:44,827 INFO nodepool.DeletedNodeWorker: Deleting used instance d6c7fb39-8f4d-47ac-9736-b8a154728484 from rdo-cloud 2018-11-29 21:13:44,831 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server d6c7fb39-8f4d-47ac-9736-b8a154728484 2018-11-29 21:13:44,834 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:13:45,026 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.19099044799804688s 2018-11-29 21:13:45,050 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkDeleteFloatingips (queue 0) 2018-11-29 21:13:45,708 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2545669078826904s 2018-11-29 21:13:45,723 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:46,835 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:13:46,907 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkDeleteFloatingips in 1.8566524982452393s 2018-11-29 21:13:47,059 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.22325897216796875s 2018-11-29 21:13:47,059 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:13:47,060 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:13:47,249 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.18937897682189941s 2018-11-29 21:13:47,253 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:47,631 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.3787217140197754s 2018-11-29 21:13:47,649 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:13:47,649 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:13:47,754 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:13:47,812 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:13:47,812 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:13:47,812 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:13:47,822 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318774 for request 300-0000331925 2018-11-29 21:13:47,831 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:13:47,831 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:13:47,832 INFO nodepool.NodeLauncher-0000318774: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318774 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318774 2018-11-29 21:13:47,833 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:13:47,922 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 2.19594669342041s 2018-11-29 21:13:48,888 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetPorts (queue 0) 2018-11-29 21:13:48,977 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.1446716785430908s 2018-11-29 21:13:48,978 ERROR nodepool.NodeLauncher-0000318774: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318774: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:13:48,978 INFO nodepool.NodeLauncher-0000318774: Quota exceeded, invalidating quota cache 2018-11-29 21:13:48,978 INFO nodepool.NodeLauncher-0000318774: Aborting node 0000318774 due to quota failure 2018-11-29 21:13:49,096 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetPorts in 0.2074112892150879s 2018-11-29 21:13:49,103 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkPostFloatingips (queue 0) 2018-11-29 21:13:49,840 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:13:49,841 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:13:49,841 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:13:49,842 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:13:49,842 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:13:49,843 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:13:50,128 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318773 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:13:50,129 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:13:50,133 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318773, state=deleting, external_id=None 2018-11-29 21:13:50,206 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:13:50,234 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:13:51,170 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:13:51,494 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkPostFloatingips in 2.3914661407470703s 2018-11-29 21:13:51,499 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 1) 2018-11-29 21:13:51,736 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:13:52,069 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:13:52,119 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:13:52,128 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:13:52,129 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:13:52,230 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: ['100-0000331950'] 2018-11-29 21:13:53,193 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4562716484069824s 2018-11-29 21:13:53,209 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:54,053 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 2.5536046028137207s 2018-11-29 21:13:54,060 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 1) 2018-11-29 21:13:54,139 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.07836508750915527s 2018-11-29 21:13:54,154 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkDeleteFloatingips (queue 0) 2018-11-29 21:13:54,293 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkDeleteFloatingips in 0.13368725776672363s 2018-11-29 21:13:54,293 ERROR nodepool.NodeDeleter: Exception deleting instance d6c7fb39-8f4d-47ac-9736-b8a154728484 from rdo-cloud: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 80, in delete manager.cleanupNode(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 445, in cleanupNode self.deleteServer(server_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 437, in deleteServer return self._client.delete_server(server_id, delete_ips=True) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 7044, in delete_server delete_ip_retry=delete_ip_retry) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 7080, in _delete_server self._delete_server_floating_ips(server, delete_ip_retry) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 7071, in _delete_server_floating_ips id=server['id'])) openstack.exceptions.SDKException: Tried to delete floating ip 38.145.33.20 associated with server d6c7fb39-8f4d-47ac-9736-b8a154728484 but there was an error deleting it. Not deleting server. 2018-11-29 21:13:54,294 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetPorts (queue 0) 2018-11-29 21:13:54,511 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetPorts in 0.21695423126220703s 2018-11-29 21:13:54,517 DEBUG nodepool.NodeLauncher-0000318772: Node 0000318772 is running [region: regionOne, az: nova, ip: 38.145.33.232 ipv4: 38.145.33.232, ipv6: ] 2018-11-29 21:13:54,518 DEBUG nodepool.NodeLauncher-0000318772: Gathering host keys for node 0000318772 2018-11-29 21:13:54,545 INFO nodepool.NodeLauncher-0000318772: Node id 0000318772 is ready 2018-11-29 21:13:55,027 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.8156871795654297s 2018-11-29 21:13:55,398 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318758 (state: deleting, allocated_to: 200-0000331949) 2018-11-29 21:13:55,399 INFO nodepool.DeletedNodeWorker: Deleting deleting instance d6c7fb39-8f4d-47ac-9736-b8a154728484 from rdo-cloud 2018-11-29 21:13:55,405 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server d6c7fb39-8f4d-47ac-9736-b8a154728484 2018-11-29 21:13:55,406 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeDeleteServers (queue 0) 2018-11-29 21:13:55,639 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeDeleteServers in 0.23262381553649902s 2018-11-29 21:13:57,836 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:13:57,969 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.1327676773071289s 2018-11-29 21:13:57,969 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:13:57,969 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:13:58,109 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13934612274169922s 2018-11-29 21:13:58,112 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:58,633 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5203731060028076s 2018-11-29 21:13:58,651 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:13:58,651 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:13:58,759 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:13:58,806 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:13:58,807 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:13:58,807 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:13:58,817 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318775 for request 300-0000331819 2018-11-29 21:13:58,824 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:13:58,824 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:13:58,825 INFO nodepool.NodeLauncher-0000318775: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318775 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318775 2018-11-29 21:13:58,826 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:13:59,222 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:13:59,649 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:13:59,962 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.1356205940246582s 2018-11-29 21:13:59,964 ERROR nodepool.NodeLauncher-0000318775: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318775: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:13:59,965 INFO nodepool.NodeLauncher-0000318775: Quota exceeded, invalidating quota cache 2018-11-29 21:13:59,965 INFO nodepool.NodeLauncher-0000318775: Aborting node 0000318775 due to quota failure 2018-11-29 21:14:00,225 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.5754754543304443s 2018-11-29 21:14:00,231 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318758, state=deleting, external_id=d6c7fb39-8f4d-47ac-9736-b8a154728484 2018-11-29 21:14:00,340 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:14:00,341 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:14:00,343 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:14:00,345 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:14:00,345 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:14:00,346 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:14:00,423 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:14:00,431 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:14:00,509 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.286832332611084s 2018-11-29 21:14:00,520 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:00,694 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318774 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:14:00,695 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:14:00,698 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318774, state=deleting, external_id=None 2018-11-29 21:14:01,282 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:14:01,516 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9959795475006104s 2018-11-29 21:14:02,183 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:14:02,269 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:14:02,282 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:14:02,283 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:14:02,338 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Fulfilled node request 100-0000331950 2018-11-29 21:14:02,342 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Unlocked node 0000318772 for request 100-0000331950 2018-11-29 21:14:02,350 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 100-0000331950 2018-11-29 21:14:02,350 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:14:02,921 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:03,308 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.3861846923828125s 2018-11-29 21:14:03,308 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-3490aba9-dbc3-4e76-9f74-5a7ba6ffc5a6) 2018-11-29 21:14:03,444 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:14:05,947 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:06,541 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:06,957 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.0094764232635498s 2018-11-29 21:14:07,851 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.310112476348877s 2018-11-29 21:14:07,866 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:14:08,829 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:14:09,035 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2055959701538086s 2018-11-29 21:14:09,035 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:14:09,036 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:14:09,176 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.14053916931152344s 2018-11-29 21:14:09,180 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:09,191 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.325425386428833s 2018-11-29 21:14:09,506 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.3257441520690918s 2018-11-29 21:14:09,524 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:14:09,525 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:14:09,645 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:14:09,690 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:14:09,690 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:14:09,690 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:14:09,700 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318776 for request 300-0000331925 2018-11-29 21:14:09,707 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:14:09,707 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:14:09,708 INFO nodepool.NodeLauncher-0000318776: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318776 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318776 2018-11-29 21:14:09,709 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:14:10,466 DEBUG nodepool.NodePool: No more active min-ready requests for label dib-dci-centos-7 2018-11-29 21:14:10,802 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:14:10,804 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:14:10,805 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:14:10,805 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:14:10,806 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:14:10,806 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:14:10,808 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:14:10,810 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:14:11,162 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318775 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:14:11,162 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:14:11,166 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318775, state=deleting, external_id=None 2018-11-29 21:14:11,366 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.657318353652954s 2018-11-29 21:14:11,367 ERROR nodepool.NodeLauncher-0000318776: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318776: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:14:11,367 INFO nodepool.NodeLauncher-0000318776: Quota exceeded, invalidating quota cache 2018-11-29 21:14:11,368 INFO nodepool.NodeLauncher-0000318776: Aborting node 0000318776 due to quota failure 2018-11-29 21:14:11,395 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:14:12,086 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:12,264 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:14:12,405 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:14:12,422 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:14:12,425 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:14:12,465 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:14:12,925 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8394491672515869s 2018-11-29 21:14:13,069 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:14,006 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.937577486038208s 2018-11-29 21:14:15,221 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:14:16,528 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3067374229431152s 2018-11-29 21:14:18,937 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:19,115 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:19,712 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:14:19,867 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.154815673828125s 2018-11-29 21:14:19,867 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:14:19,868 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:14:19,879 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9422433376312256s 2018-11-29 21:14:20,053 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.18568921089172363s 2018-11-29 21:14:20,057 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:20,307 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1920008659362793s 2018-11-29 21:14:20,687 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6299777030944824s 2018-11-29 21:14:20,705 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:14:20,706 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:14:21,300 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:14:21,348 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:14:21,349 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:14:21,350 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:14:21,350 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:14:21,350 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:14:21,351 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:14:21,352 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:14:21,353 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:14:21,420 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:14:21,420 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:14:21,420 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:14:21,431 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318777 for request 300-0000331819 2018-11-29 21:14:21,441 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:14:21,441 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:14:21,443 INFO nodepool.NodeLauncher-0000318777: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318777 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318777 2018-11-29 21:14:21,444 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:14:21,565 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:14:21,703 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318776 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:14:21,703 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:14:21,706 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318776, state=deleting, external_id=None 2018-11-29 21:14:22,352 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:14:22,557 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:14:22,560 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:14:22,590 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:14:22,593 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:14:22,615 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:14:22,845 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4007391929626465s 2018-11-29 21:14:22,845 ERROR nodepool.NodeLauncher-0000318777: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318777: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:14:22,846 INFO nodepool.NodeLauncher-0000318777: Quota exceeded, invalidating quota cache 2018-11-29 21:14:22,846 INFO nodepool.NodeLauncher-0000318777: Aborting node 0000318777 due to quota failure 2018-11-29 21:14:23,954 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3967294692993164s 2018-11-29 21:14:24,995 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:25,559 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:25,773 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7772953510284424s 2018-11-29 21:14:26,559 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0001232624053955s 2018-11-29 21:14:26,966 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318748 (state: used, allocated_to: 200-0000331943) 2018-11-29 21:14:26,966 INFO nodepool.DeletedNodeWorker: Deleting used instance 7779798d-36d4-40d1-9e91-29b211616bf8 from rdo-cloud 2018-11-29 21:14:26,978 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 7779798d-36d4-40d1-9e91-29b211616bf8 2018-11-29 21:14:26,981 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:14:27,347 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.36510205268859863s 2018-11-29 21:14:27,370 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkDeleteFloatingips (queue 0) 2018-11-29 21:14:28,553 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkDeleteFloatingips in 1.1821532249450684s 2018-11-29 21:14:29,983 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:14:31,162 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1786181926727295s 2018-11-29 21:14:31,622 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:14:31,643 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:31,760 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.13738083839416504s 2018-11-29 21:14:31,760 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:14:31,761 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:14:31,786 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:31,951 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1898815631866455s 2018-11-29 21:14:31,954 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:32,071 INFO nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Assigning node request 2018-11-29 21:14:32,071 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-dfw-1-s1.large]: Declining node request 200-0000331951 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:14:32,084 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:14:32,085 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:14:32,085 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:14:32,087 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:14:32,087 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:14:32,087 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:14:32,088 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:14:32,089 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:14:32,108 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Removing handler for request 200-0000331951 2018-11-29 21:14:32,108 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:14:32,391 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6042296886444092s 2018-11-29 21:14:32,400 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.4454939365386963s 2018-11-29 21:14:32,420 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:14:32,420 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:14:32,421 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.7775547504425049s 2018-11-29 21:14:32,481 INFO nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Assigning node request 2018-11-29 21:14:32,482 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.limestone-ansible-network-us-slc-s1.large]: Declining node request 200-0000331951 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:14:32,499 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Removing handler for request 200-0000331951 2018-11-29 21:14:32,499 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:14:32,577 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:14:32,737 INFO nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Assigning node request 2018-11-29 21:14:32,737 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io]: Declining node request 200-0000331951 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:14:32,752 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:14:32,752 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:14:32,752 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:14:32,769 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318778 for request 300-0000331925 2018-11-29 21:14:32,784 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:14:32,784 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:14:32,785 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Removing handler for request 200-0000331951 2018-11-29 21:14:32,785 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:14:32,789 INFO nodepool.NodeLauncher-0000318778: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318778 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318778 2018-11-29 21:14:32,790 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:14:32,815 INFO nodepool.PoolWorker.rdo-cloud-rdoproject.org: Assigning node request 2018-11-29 21:14:32,817 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-rdoproject.org]: Declining node request 200-0000331951 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:14:32,820 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:14:32,835 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Removing handler for request 200-0000331951 2018-11-29 21:14:32,835 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:14:32,835 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:14:33,559 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:14:33,670 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.1107783317565918s 2018-11-29 21:14:33,686 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.8961031436920166s 2018-11-29 21:14:33,694 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeDeleteServers (queue 0) 2018-11-29 21:14:33,687 ERROR nodepool.NodeLauncher-0000318778: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318778: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:14:33,699 INFO nodepool.NodeLauncher-0000318778: Quota exceeded, invalidating quota cache 2018-11-29 21:14:33,700 INFO nodepool.NodeLauncher-0000318778: Aborting node 0000318778 due to quota failure 2018-11-29 21:14:33,933 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeDeleteServers in 0.23863458633422852s 2018-11-29 21:14:33,935 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetPorts (queue 0) 2018-11-29 21:14:34,132 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetPorts in 0.197343111038208s 2018-11-29 21:14:37,194 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:14:37,490 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318777 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:14:37,490 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:14:37,495 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318777, state=deleting, external_id=None 2018-11-29 21:14:37,915 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:38,534 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3394873142242432s 2018-11-29 21:14:38,544 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:38,642 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7271671295166016s 2018-11-29 21:14:39,418 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8683581352233887s 2018-11-29 21:14:40,144 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318748, state=deleting, external_id=7779798d-36d4-40d1-9e91-29b211616bf8 2018-11-29 21:14:42,412 ERROR nodepool.NodeDeleter: Exception deleting instance 08395f7e-438a-42c5-b393-747dc95db233 from rdo-cloud: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 08395f7e-438a-42c5-b393-747dc95db233 deletion 2018-11-29 21:14:42,637 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:14:42,663 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:14:42,665 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 200-0000331951 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:14:42,670 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:14:42,672 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:14:42,674 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:14:42,675 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:14:42,683 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:14:42,686 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:14:42,688 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:14:42,704 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 200-0000331951 2018-11-29 21:14:42,704 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:14:42,797 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:14:42,823 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:14:42,974 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000242265 (state: deleting, allocated_to: None) 2018-11-29 21:14:42,975 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 08395f7e-438a-42c5-b393-747dc95db233 from rdo-cloud 2018-11-29 21:14:42,981 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 08395f7e-438a-42c5-b393-747dc95db233 2018-11-29 21:14:42,982 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeDeleteServers (queue 0) 2018-11-29 21:14:43,012 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:14:43,042 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:14:43,042 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Assigning node request 2018-11-29 21:14:43,045 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1]: Declining node request 200-0000331951 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:14:43,048 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeDeleteServers in 0.06545114517211914s 2018-11-29 21:14:43,054 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:14:43,065 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Removing handler for request 200-0000331951 2018-11-29 21:14:43,065 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:14:43,219 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.4218566417694092s 2018-11-29 21:14:43,220 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:14:43,220 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:14:43,423 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.20318102836608887s 2018-11-29 21:14:43,427 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:44,004 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5772609710693359s 2018-11-29 21:14:44,023 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:14:44,024 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:14:44,147 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:14:44,200 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:14:44,200 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:14:44,200 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:14:44,212 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318779 for request 300-0000331819 2018-11-29 21:14:44,219 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:14:44,219 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:14:44,220 INFO nodepool.NodeLauncher-0000318779: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318779 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318779 2018-11-29 21:14:44,221 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:14:44,454 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:44,655 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:45,208 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.5525729656219482s 2018-11-29 21:14:45,428 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2073683738708496s 2018-11-29 21:14:45,429 ERROR nodepool.NodeLauncher-0000318779: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318779: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:14:45,429 INFO nodepool.NodeLauncher-0000318779: Quota exceeded, invalidating quota cache 2018-11-29 21:14:45,429 INFO nodepool.NodeLauncher-0000318779: Aborting node 0000318779 due to quota failure 2018-11-29 21:14:45,671 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2165024280548096s 2018-11-29 21:14:45,688 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:14:46,992 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3039255142211914s 2018-11-29 21:14:48,182 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318778 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:14:48,182 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:14:48,187 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318778, state=deleting, external_id=None 2018-11-29 21:14:51,016 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:51,062 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:51,809 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.7929747104644775s 2018-11-29 21:14:51,863 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8013997077941895s 2018-11-29 21:14:53,246 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:14:53,264 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:14:53,266 INFO nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Assigning node request 2018-11-29 21:14:53,266 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4]: Declining node request 200-0000331951 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:14:53,285 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:14:53,285 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:14:53,286 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:14:53,289 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:14:53,290 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:14:53,290 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:14:53,322 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Removing handler for request 200-0000331951 2018-11-29 21:14:53,322 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:14:53,332 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:14:53,446 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:14:53,471 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:14:53,475 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:14:53,481 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetLimits (queue 0) 2018-11-29 21:14:53,484 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:14:53,524 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetLimits in 0.04237675666809082s 2018-11-29 21:14:53,524 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for rdo-cloud: {'compute': {'cores': 520, 'ram': 532480, 'instances': 65}} 2018-11-29 21:14:53,525 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:14:53,582 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetFlavorsDetail in 0.05697274208068848s 2018-11-29 21:14:53,585 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for rdo-cloud: {'compute': {'cores': 519, 'ram': 530432, 'instances': 64}} 2018-11-29 21:14:53,585 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Accepting node request 200-0000331951 2018-11-29 21:14:53,676 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Locked existing node 0000318772 for request 200-0000331951 2018-11-29 21:14:53,680 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Fulfilled node request 200-0000331951 2018-11-29 21:14:53,683 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Unlocked node 0000318772 for request 200-0000331951 2018-11-29 21:14:53,688 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 200-0000331951 2018-11-29 21:14:53,689 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:14:53,826 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:14:54,224 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:14:54,356 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.13177919387817383s 2018-11-29 21:14:54,356 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:14:54,357 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:14:54,485 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1286478042602539s 2018-11-29 21:14:54,489 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:54,901 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.41191744804382324s 2018-11-29 21:14:54,919 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:14:54,920 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:14:55,026 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:14:55,074 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:14:55,074 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:14:55,074 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:14:55,084 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318780 for request 300-0000331925 2018-11-29 21:14:55,091 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:14:55,091 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:14:55,093 INFO nodepool.NodeLauncher-0000318780: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318780 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318780 2018-11-29 21:14:55,093 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:14:55,171 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3441808223724365s 2018-11-29 21:14:56,266 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.1726233959197998s 2018-11-29 21:14:56,266 ERROR nodepool.NodeLauncher-0000318780: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318780: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:14:56,267 INFO nodepool.NodeLauncher-0000318780: Quota exceeded, invalidating quota cache 2018-11-29 21:14:56,267 INFO nodepool.NodeLauncher-0000318780: Aborting node 0000318780 due to quota failure 2018-11-29 21:14:57,191 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:57,233 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:14:58,069 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8774890899658203s 2018-11-29 21:14:58,347 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.1143121719360352s 2018-11-29 21:14:58,800 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318779 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:14:58,800 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:14:58,804 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318779, state=deleting, external_id=None 2018-11-29 21:15:02,091 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:15:03,413 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.321432113647461s 2018-11-29 21:15:03,433 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:03,885 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:03,961 INFO nodepool.NodePool: Creating requests for 1 dib-dci-centos-7 nodes 2018-11-29 21:15:03,988 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:15:03,991 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:15:03,994 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:15:03,995 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:15:03,995 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:15:03,996 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:15:03,997 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:15:04,000 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:15:04,000 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:15:04,002 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:15:04,050 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:15:04,057 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:15:04,064 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:15:04,137 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:15:04,520 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:04,846 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9605891704559326s 2018-11-29 21:15:04,854 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.33333420753479004s 2018-11-29 21:15:04,854 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-53a7759a-91aa-44f4-b64f-b5460d6dc96c) 2018-11-29 21:15:04,988 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.5551109313964844s 2018-11-29 21:15:05,007 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:15:05,096 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:15:05,361 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.26496100425720215s 2018-11-29 21:15:05,362 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:15:05,362 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:15:05,606 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.24321651458740234s 2018-11-29 21:15:05,609 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:06,090 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.4806396961212158s 2018-11-29 21:15:06,108 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:15:06,109 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:15:06,245 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:15:06,301 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:15:06,301 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:15:06,301 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:15:06,311 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318781 for request 300-0000331819 2018-11-29 21:15:06,319 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:15:06,319 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:15:06,320 INFO nodepool.NodeLauncher-0000318781: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318781 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318781 2018-11-29 21:15:06,321 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:15:08,180 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.8587865829467773s 2018-11-29 21:15:08,180 ERROR nodepool.NodeLauncher-0000318781: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318781: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:15:08,182 INFO nodepool.NodeLauncher-0000318781: Quota exceeded, invalidating quota cache 2018-11-29 21:15:08,182 INFO nodepool.NodeLauncher-0000318781: Aborting node 0000318781 due to quota failure 2018-11-29 21:15:09,420 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318780 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:15:09,420 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:15:09,426 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318780, state=deleting, external_id=None 2018-11-29 21:15:09,443 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:15:10,369 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:10,623 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1794447898864746s 2018-11-29 21:15:10,640 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:10,983 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6131992340087891s 2018-11-29 21:15:11,860 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2198550701141357s 2018-11-29 21:15:14,784 INFO nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Assigning node request 2018-11-29 21:15:14,796 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:15:14,797 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:15:14,797 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:15:14,798 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:15:14,798 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:15:14,799 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:15:14,800 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:15:14,802 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:15:14,802 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:15:14,802 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:15:14,802 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:15:14,802 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:15:14,809 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:15:14,971 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4]: Declining node request 100-0000331952 because node type(s) [dib-dci-centos-7] not available 2018-11-29 21:15:14,986 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Removing handler for request 100-0000331952 2018-11-29 21:15:14,987 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:15:16,323 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:15:16,586 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.26232242584228516s 2018-11-29 21:15:16,586 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:15:16,587 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:15:16,656 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:15:16,812 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.2246537208557129s 2018-11-29 21:15:16,815 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:16,870 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:17,641 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.8262193202972412s 2018-11-29 21:15:17,660 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:15:17,660 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:15:17,711 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8402776718139648s 2018-11-29 21:15:17,789 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:15:17,844 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:15:17,844 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:15:17,844 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:15:17,855 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318782 for request 300-0000331925 2018-11-29 21:15:17,862 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:15:17,862 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:15:17,863 INFO nodepool.NodeLauncher-0000318782: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318782 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318782 2018-11-29 21:15:17,864 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:15:18,013 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3547914028167725s 2018-11-29 21:15:18,029 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:19,206 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.342545747756958s 2018-11-29 21:15:19,207 ERROR nodepool.NodeLauncher-0000318782: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318782: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:15:19,207 INFO nodepool.NodeLauncher-0000318782: Quota exceeded, invalidating quota cache 2018-11-29 21:15:19,207 INFO nodepool.NodeLauncher-0000318782: Aborting node 0000318782 due to quota failure 2018-11-29 21:15:19,581 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.5486457347869873s 2018-11-29 21:15:20,170 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318781 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:15:20,170 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:15:20,176 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318781, state=deleting, external_id=None 2018-11-29 21:15:23,008 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:23,901 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8926339149475098s 2018-11-29 21:15:24,042 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:15:25,378 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.335606575012207s 2018-11-29 21:15:25,395 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:25,588 INFO nodepool.PoolWorker.rdo-cloud-main: Assigning node request 2018-11-29 21:15:25,601 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:15:25,605 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:15:25,611 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:15:25,612 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:15:25,612 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:15:25,613 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:15:25,613 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:15:25,614 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:15:25,615 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:15:25,616 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:15:25,616 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:15:25,619 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:15:25,632 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:15:25,773 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Accepting node request 100-0000331952 2018-11-29 21:15:25,869 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:15:25,886 WARNING nodepool.driver.openstack.OpenStackProvider: Cannot find provider pool for node 2018-11-29 21:15:25,897 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Predicted remaining provider quota: {'compute': {'cores': 435, 'ram': 399360, 'instances': 48}} 2018-11-29 21:15:25,956 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 15}} 2018-11-29 21:15:25,956 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 14}} 2018-11-29 21:15:25,965 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Locked building node 0000318783 for request 100-0000331952 2018-11-29 21:15:25,971 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: ['100-0000331952'] 2018-11-29 21:15:25,972 INFO nodepool.NodeLauncher-0000318783: Creating server with hostname dib-dci-centos-7-rdo-cloud-0000318783 in rdo-cloud from image dib-dci-centos-7 for node id: 0000318783 2018-11-29 21:15:25,972 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputePostServers (queue 0) 2018-11-29 21:15:26,599 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2038178443908691s 2018-11-29 21:15:27,633 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputePostServers in 1.6608140468597412s 2018-11-29 21:15:27,634 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServers (queue 0) 2018-11-29 21:15:27,866 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:15:27,940 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServers in 0.3061995506286621s 2018-11-29 21:15:27,945 DEBUG nodepool.NodeLauncher-0000318783: Waiting for server c4fc3033-0b43-49d7-82de-cbb196ece79d for node id: 0000318783 2018-11-29 21:15:28,061 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.194688081741333s 2018-11-29 21:15:28,061 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:15:28,062 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:15:28,314 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.2520439624786377s 2018-11-29 21:15:28,317 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:28,873 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5552644729614258s 2018-11-29 21:15:28,890 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:15:28,890 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:15:29,000 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:15:29,052 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:15:29,052 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:15:29,052 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:15:29,062 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318784 for request 300-0000331819 2018-11-29 21:15:29,068 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:15:29,069 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:15:29,070 INFO nodepool.NodeLauncher-0000318784: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318784 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318784 2018-11-29 21:15:29,071 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:15:29,733 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:29,999 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.9284274578094482s 2018-11-29 21:15:30,000 ERROR nodepool.NodeLauncher-0000318784: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318784: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:15:30,000 INFO nodepool.NodeLauncher-0000318784: Quota exceeded, invalidating quota cache 2018-11-29 21:15:30,001 INFO nodepool.NodeLauncher-0000318784: Aborting node 0000318784 due to quota failure 2018-11-29 21:15:30,149 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.4159839153289795s 2018-11-29 21:15:30,940 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318782 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:15:30,940 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:15:30,943 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318782, state=deleting, external_id=None 2018-11-29 21:15:31,407 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:15:32,646 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.238628625869751s 2018-11-29 21:15:32,662 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:34,017 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.352935791015625s 2018-11-29 21:15:35,922 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:36,356 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:15:36,357 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:15:36,357 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:15:36,358 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:15:36,359 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:15:36,362 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:15:36,363 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:15:36,363 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:15:36,364 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:15:36,366 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:15:36,366 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:15:36,368 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:15:36,368 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:15:36,468 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: ['100-0000331952'] 2018-11-29 21:15:36,749 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8262026309967041s 2018-11-29 21:15:37,959 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetPorts (queue 0) 2018-11-29 21:15:38,282 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetPorts in 0.32193899154663086s 2018-11-29 21:15:38,287 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:15:38,388 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.1013491153717041s 2018-11-29 21:15:38,413 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkPostFloatingips (queue 0) 2018-11-29 21:15:38,675 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:15:39,073 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:15:39,260 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.18697667121887207s 2018-11-29 21:15:39,260 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:15:39,261 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:15:39,399 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1383202075958252s 2018-11-29 21:15:39,402 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:39,773 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.3708202838897705s 2018-11-29 21:15:39,794 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:15:39,794 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:15:39,889 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:15:39,936 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:15:39,937 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:15:39,937 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:15:39,946 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318785 for request 300-0000331925 2018-11-29 21:15:39,954 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:15:39,954 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:15:39,955 INFO nodepool.NodeLauncher-0000318785: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318785 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318785 2018-11-29 21:15:39,956 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:15:39,982 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3069450855255127s 2018-11-29 21:15:39,998 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:41,001 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkPostFloatingips in 2.5878522396087646s 2018-11-29 21:15:41,085 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.1287527084350586s 2018-11-29 21:15:41,085 ERROR nodepool.NodeLauncher-0000318785: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318785: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:15:41,085 INFO nodepool.NodeLauncher-0000318785: Quota exceeded, invalidating quota cache 2018-11-29 21:15:41,086 INFO nodepool.NodeLauncher-0000318785: Aborting node 0000318785 due to quota failure 2018-11-29 21:15:41,103 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1045422554016113s 2018-11-29 21:15:41,724 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318784 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:15:41,724 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:15:41,729 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318784, state=deleting, external_id=None 2018-11-29 21:15:42,171 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:44,110 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.9387986660003662s 2018-11-29 21:15:46,011 DEBUG nodepool.TaskManager: Manager rdo-cloud running task NetworkGetFloatingips (queue 0) 2018-11-29 21:15:46,021 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:15:46,077 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task NetworkGetFloatingips in 0.06587934494018555s 2018-11-29 21:15:46,107 DEBUG nodepool.NodeLauncher-0000318783: Node 0000318783 is running [region: regionOne, az: nova, ip: 38.145.32.36 ipv4: 38.145.32.36, ipv6: ] 2018-11-29 21:15:46,107 DEBUG nodepool.NodeLauncher-0000318783: Gathering host keys for node 0000318783 2018-11-29 21:15:46,133 INFO nodepool.NodeLauncher-0000318783: Node id 0000318783 is ready 2018-11-29 21:15:47,106 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:15:47,108 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:15:47,108 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:15:47,109 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:15:47,109 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:15:47,110 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:15:47,110 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:15:47,111 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:15:47,111 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:15:47,111 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:15:47,113 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:15:47,113 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:15:47,115 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:15:47,125 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Fulfilled node request 100-0000331952 2018-11-29 21:15:47,127 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.rdo-cloud-main]: Unlocked node 0000318783 for request 100-0000331952 2018-11-29 21:15:47,138 DEBUG nodepool.PoolWorker.rdo-cloud-main: Removing handler for request 100-0000331952 2018-11-29 21:15:47,138 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:15:47,931 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.9101331233978271s 2018-11-29 21:15:47,947 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:49,230 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.280597448348999s 2018-11-29 21:15:49,958 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:15:50,090 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.1316850185394287s 2018-11-29 21:15:50,090 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:15:50,091 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:15:50,125 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:50,226 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13491439819335938s 2018-11-29 21:15:50,230 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:50,570 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.34038782119750977s 2018-11-29 21:15:50,588 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:15:50,589 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:15:50,721 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:15:50,789 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:15:50,789 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:15:50,789 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:15:50,800 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318786 for request 300-0000331819 2018-11-29 21:15:50,809 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:15:50,809 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:15:50,810 INFO nodepool.NodeLauncher-0000318786: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318786 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318786 2018-11-29 21:15:50,811 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:15:51,122 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9963915348052979s 2018-11-29 21:15:51,601 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.7898638248443604s 2018-11-29 21:15:51,601 ERROR nodepool.NodeLauncher-0000318786: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318786: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:15:51,602 INFO nodepool.NodeLauncher-0000318786: Quota exceeded, invalidating quota cache 2018-11-29 21:15:51,602 INFO nodepool.NodeLauncher-0000318786: Aborting node 0000318786 due to quota failure 2018-11-29 21:15:53,413 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318785 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:15:53,413 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:15:53,417 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318785, state=deleting, external_id=None 2018-11-29 21:15:53,966 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:15:55,176 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2096247673034668s 2018-11-29 21:15:55,192 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:56,047 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8546943664550781s 2018-11-29 21:15:56,783 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:15:57,216 DEBUG nodepool.NodePool: No more active min-ready requests for label dib-dci-centos-7 2018-11-29 21:15:57,520 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7359747886657715s 2018-11-29 21:15:57,848 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:15:57,849 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:15:57,850 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:15:57,851 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:15:57,851 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:15:57,851 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:15:57,852 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:15:57,853 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:15:57,854 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:15:57,854 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:15:57,854 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:15:57,854 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:15:57,855 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:15:57,855 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:16:00,814 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:16:01,072 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2583327293395996s 2018-11-29 21:16:01,073 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:16:01,073 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:16:01,148 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:01,315 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.24171996116638184s 2018-11-29 21:16:01,319 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:01,904 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5857555866241455s 2018-11-29 21:16:01,922 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:16:01,922 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:16:02,058 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:16:02,117 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:16:02,117 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:16:02,117 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:16:02,128 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318787 for request 300-0000331925 2018-11-29 21:16:02,135 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:16:02,136 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:16:02,137 INFO nodepool.NodeLauncher-0000318787: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318787 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318787 2018-11-29 21:16:02,138 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:16:02,193 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0442371368408203s 2018-11-29 21:16:02,206 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:16:03,157 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:03,677 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.5390677452087402s 2018-11-29 21:16:03,677 ERROR nodepool.NodeLauncher-0000318787: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318787: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:16:03,678 INFO nodepool.NodeLauncher-0000318787: Quota exceeded, invalidating quota cache 2018-11-29 21:16:03,678 INFO nodepool.NodeLauncher-0000318787: Aborting node 0000318787 due to quota failure 2018-11-29 21:16:03,687 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4808316230773926s 2018-11-29 21:16:03,884 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318786 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:16:03,884 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:16:03,887 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318786, state=deleting, external_id=None 2018-11-29 21:16:03,918 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.760906457901001s 2018-11-29 21:16:05,415 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:05,761 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.34577059745788574s 2018-11-29 21:16:05,762 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f4a4c8e2-e7f4-4bc0-9a28-b724a2682cb2) 2018-11-29 21:16:05,904 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:16:07,274 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:08,248 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9739108085632324s 2018-11-29 21:16:08,595 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:16:08,596 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:16:08,599 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:16:08,600 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:16:08,600 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:16:08,602 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:16:08,602 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:16:08,603 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:16:08,603 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:16:08,603 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:16:08,603 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:16:08,604 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:16:08,604 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:16:08,604 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:16:09,543 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:09,717 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:16:10,283 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7398831844329834s 2018-11-29 21:16:11,162 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4451494216918945s 2018-11-29 21:16:12,141 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:16:12,355 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.21377277374267578s 2018-11-29 21:16:12,356 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:16:12,356 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:16:12,538 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.18160271644592285s 2018-11-29 21:16:12,541 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:13,097 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5552229881286621s 2018-11-29 21:16:13,115 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:16:13,116 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:16:13,240 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:16:13,307 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:16:13,307 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:16:13,307 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:16:13,318 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318788 for request 300-0000331819 2018-11-29 21:16:13,324 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:16:13,325 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:16:13,326 INFO nodepool.NodeLauncher-0000318788: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318788 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318788 2018-11-29 21:16:13,327 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:16:13,686 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:14,377 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318787 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:16:14,377 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:16:14,381 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318787, state=deleting, external_id=None 2018-11-29 21:16:14,632 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.945274829864502s 2018-11-29 21:16:14,675 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.3476102352142334s 2018-11-29 21:16:14,675 ERROR nodepool.NodeLauncher-0000318788: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318788: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:16:14,675 INFO nodepool.NodeLauncher-0000318788: Quota exceeded, invalidating quota cache 2018-11-29 21:16:14,676 INFO nodepool.NodeLauncher-0000318788: Aborting node 0000318788 due to quota failure 2018-11-29 21:16:15,944 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:16,813 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8684976100921631s 2018-11-29 21:16:17,191 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:16:18,504 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3128983974456787s 2018-11-29 21:16:19,403 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:16:19,404 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:16:19,405 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:16:19,407 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:16:19,408 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:16:19,408 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:16:19,409 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:16:19,409 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:16:19,409 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:16:19,410 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:16:19,410 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:16:19,411 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:16:19,411 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:16:19,411 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:16:19,856 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:20,819 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9629828929901123s 2018-11-29 21:16:22,305 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:22,933 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6269710063934326s 2018-11-29 21:16:23,329 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:16:23,563 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.23392105102539062s 2018-11-29 21:16:23,563 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:16:23,564 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:16:23,694 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13078594207763672s 2018-11-29 21:16:23,698 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:24,531 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:16:24,681 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.9835758209228516s 2018-11-29 21:16:24,699 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:16:24,699 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:16:24,965 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:16:25,038 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:16:25,039 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:16:25,039 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:16:25,078 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318789 for request 300-0000331925 2018-11-29 21:16:25,096 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:16:25,096 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:16:25,098 INFO nodepool.NodeLauncher-0000318789: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318789 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318789 2018-11-29 21:16:25,098 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:16:25,823 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2916224002838135s 2018-11-29 21:16:25,978 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:26,995 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0170872211456299s 2018-11-29 21:16:27,120 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 2.0218422412872314s 2018-11-29 21:16:27,121 ERROR nodepool.NodeLauncher-0000318789: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318789: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:16:27,121 INFO nodepool.NodeLauncher-0000318789: Quota exceeded, invalidating quota cache 2018-11-29 21:16:27,121 INFO nodepool.NodeLauncher-0000318789: Aborting node 0000318789 due to quota failure 2018-11-29 21:16:28,255 ERROR nodepool.NodeDeleter: Exception deleting instance b8f745b0-ef71-49e0-b432-3868667292b9 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server b8f745b0-ef71-49e0-b432-3868667292b9 deletion 2018-11-29 21:16:28,838 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:29,518 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6799800395965576s 2018-11-29 21:16:30,160 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:16:30,161 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:16:30,162 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:16:30,163 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:16:30,165 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:16:30,166 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:16:30,167 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:16:30,167 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:16:30,168 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:16:30,169 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:16:30,169 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:16:30,170 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:16:30,171 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:16:30,171 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:16:30,178 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211292 (state: deleting, allocated_to: None) 2018-11-29 21:16:30,178 INFO nodepool.DeletedNodeWorker: Deleting deleting instance b8f745b0-ef71-49e0-b432-3868667292b9 from rdo-cloud-tripleo 2018-11-29 21:16:30,183 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server b8f745b0-ef71-49e0-b432-3868667292b9 2018-11-29 21:16:30,184 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:16:30,266 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.08175826072692871s 2018-11-29 21:16:30,312 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318788 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:16:30,312 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:16:30,317 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318788, state=deleting, external_id=None 2018-11-29 21:16:31,852 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:16:33,401 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.5486323833465576s 2018-11-29 21:16:33,417 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:34,543 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1225500106811523s 2018-11-29 21:16:34,958 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:35,102 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:16:35,263 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.16086530685424805s 2018-11-29 21:16:35,264 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:16:35,265 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:16:35,472 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.20734858512878418s 2018-11-29 21:16:35,476 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:35,849 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8910789489746094s 2018-11-29 21:16:36,052 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5757279396057129s 2018-11-29 21:16:36,070 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:16:36,071 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:16:36,183 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:16:36,236 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:16:36,236 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:16:36,237 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:16:36,778 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318790 for request 300-0000331819 2018-11-29 21:16:36,804 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:16:36,804 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:16:36,806 INFO nodepool.NodeLauncher-0000318790: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318790 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318790 2018-11-29 21:16:36,807 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:16:38,327 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.5196771621704102s 2018-11-29 21:16:38,327 ERROR nodepool.NodeLauncher-0000318790: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318790: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:16:38,328 INFO nodepool.NodeLauncher-0000318790: Quota exceeded, invalidating quota cache 2018-11-29 21:16:38,328 INFO nodepool.NodeLauncher-0000318790: Aborting node 0000318790 due to quota failure 2018-11-29 21:16:39,434 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:16:40,961 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.527400255203247s 2018-11-29 21:16:40,982 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:41,050 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:16:41,052 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:16:41,052 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:16:41,052 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:16:41,053 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:16:41,054 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:16:41,057 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:16:41,057 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:16:41,058 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:16:41,058 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:16:41,058 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:16:41,058 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:16:41,059 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:16:41,059 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:16:41,184 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318789 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:16:41,184 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:16:41,188 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318789, state=deleting, external_id=None 2018-11-29 21:16:41,541 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:42,187 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6453814506530762s 2018-11-29 21:16:42,251 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2687294483184814s 2018-11-29 21:16:46,809 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:16:46,991 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:16:47,010 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.20008158683776855s 2018-11-29 21:16:47,010 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:16:47,011 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:16:47,157 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1462411880493164s 2018-11-29 21:16:47,160 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:47,479 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.3182649612426758s 2018-11-29 21:16:47,498 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:16:47,498 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:16:47,595 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:16:47,644 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:16:47,644 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:16:47,644 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:16:47,655 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318791 for request 300-0000331925 2018-11-29 21:16:47,662 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:16:47,662 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:16:47,664 INFO nodepool.NodeLauncher-0000318791: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318791 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318791 2018-11-29 21:16:47,665 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:16:47,871 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:48,332 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3401963710784912s 2018-11-29 21:16:48,347 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:48,880 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.0082073211669922s 2018-11-29 21:16:48,979 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.3139808177947998s 2018-11-29 21:16:48,979 ERROR nodepool.NodeLauncher-0000318791: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318791: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:16:48,981 INFO nodepool.NodeLauncher-0000318791: Quota exceeded, invalidating quota cache 2018-11-29 21:16:48,981 INFO nodepool.NodeLauncher-0000318791: Aborting node 0000318791 due to quota failure 2018-11-29 21:16:49,194 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8434543609619141s 2018-11-29 21:16:51,756 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:16:51,758 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:16:51,758 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:16:51,758 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:16:51,759 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:16:51,759 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:16:51,762 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:16:51,762 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:16:51,762 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:16:51,763 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:16:51,763 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:16:51,764 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:16:51,765 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:16:51,765 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:16:52,765 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318790 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:16:52,765 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:16:52,768 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318790, state=deleting, external_id=None 2018-11-29 21:16:54,216 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:54,293 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:54,855 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6385548114776611s 2018-11-29 21:16:55,221 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.92757248878479s 2018-11-29 21:16:55,234 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:16:56,959 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.7255840301513672s 2018-11-29 21:16:57,669 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:16:57,842 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.17231059074401855s 2018-11-29 21:16:57,842 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:16:57,842 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:16:58,042 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.19928908348083496s 2018-11-29 21:16:58,045 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:16:58,378 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.33290576934814453s 2018-11-29 21:16:58,396 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:16:58,396 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:16:58,511 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:16:58,565 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:16:58,565 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:16:58,566 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:16:58,577 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318792 for request 300-0000331819 2018-11-29 21:16:58,585 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:16:58,585 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:16:58,588 INFO nodepool.NodeLauncher-0000318792: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318792 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318792 2018-11-29 21:16:58,588 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:16:59,653 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.0642681121826172s 2018-11-29 21:16:59,653 ERROR nodepool.NodeLauncher-0000318792: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318792: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:16:59,654 INFO nodepool.NodeLauncher-0000318792: Quota exceeded, invalidating quota cache 2018-11-29 21:16:59,654 INFO nodepool.NodeLauncher-0000318792: Aborting node 0000318792 due to quota failure 2018-11-29 21:17:00,310 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:00,866 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:01,287 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.977083683013916s 2018-11-29 21:17:01,550 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6841428279876709s 2018-11-29 21:17:02,457 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:17:02,458 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:17:02,459 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:17:02,459 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:17:02,459 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:17:02,461 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:17:02,462 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:17:02,462 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:17:02,462 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:17:02,463 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:17:02,463 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:17:02,464 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:17:02,464 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:17:02,464 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:17:02,989 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:17:03,224 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318791 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:17:03,224 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:17:03,230 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318791, state=deleting, external_id=None 2018-11-29 21:17:04,306 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3171889781951904s 2018-11-29 21:17:06,300 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:06,325 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:06,655 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.35442376136779785s 2018-11-29 21:17:06,655 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-5cdca5d2-cd69-444a-bfa3-819c64bd3220) 2018-11-29 21:17:06,799 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:17:06,911 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:07,334 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0093247890472412s 2018-11-29 21:17:07,361 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.4494435787200928s 2018-11-29 21:17:08,589 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:17:08,795 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.20609188079833984s 2018-11-29 21:17:08,796 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:17:08,796 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:17:09,080 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.28341054916381836s 2018-11-29 21:17:09,083 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:09,844 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.7604751586914062s 2018-11-29 21:17:09,863 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:17:09,864 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:17:09,995 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:17:10,060 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:17:10,060 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:17:10,060 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:17:10,072 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318793 for request 300-0000331925 2018-11-29 21:17:10,080 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:17:10,080 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:17:10,082 INFO nodepool.NodeLauncher-0000318793: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318793 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318793 2018-11-29 21:17:10,082 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:17:10,337 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:17:11,263 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.1808362007141113s 2018-11-29 21:17:11,263 ERROR nodepool.NodeLauncher-0000318793: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318793: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:17:11,264 INFO nodepool.NodeLauncher-0000318793: Quota exceeded, invalidating quota cache 2018-11-29 21:17:11,264 INFO nodepool.NodeLauncher-0000318793: Aborting node 0000318793 due to quota failure 2018-11-29 21:17:11,619 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.281036138534546s 2018-11-29 21:17:12,364 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:13,266 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:17:13,270 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:17:13,271 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:17:13,271 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:17:13,272 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:17:13,273 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:17:13,274 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:17:13,274 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:17:13,274 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:17:13,275 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:17:13,275 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:17:13,276 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:17:13,276 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:17:13,276 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:17:13,374 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:13,720 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318792 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:17:13,720 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:17:13,723 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318792, state=deleting, external_id=None 2018-11-29 21:17:13,835 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.470717430114746s 2018-11-29 21:17:14,218 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8439371585845947s 2018-11-29 21:17:16,903 ERROR nodepool.NodeDeleter: Exception deleting instance 125c7ce6-fd70-4a46-85c8-b6728da772e5 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 125c7ce6-fd70-4a46-85c8-b6728da772e5 deletion 2018-11-29 21:17:17,650 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:17:18,981 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3306221961975098s 2018-11-29 21:17:19,000 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:19,003 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211296 (state: deleting, allocated_to: None) 2018-11-29 21:17:19,003 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 125c7ce6-fd70-4a46-85c8-b6728da772e5 from rdo-cloud-tripleo 2018-11-29 21:17:19,007 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 125c7ce6-fd70-4a46-85c8-b6728da772e5 2018-11-29 21:17:19,586 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:20,084 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:17:20,279 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.693352460861206s 2018-11-29 21:17:20,288 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2037365436553955s 2018-11-29 21:17:20,289 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:17:20,289 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:17:20,293 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2927725315093994s 2018-11-29 21:17:20,307 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:17:20,384 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.0773308277130127s 2018-11-29 21:17:20,505 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.2158961296081543s 2018-11-29 21:17:20,508 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:21,059 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5502846240997314s 2018-11-29 21:17:21,078 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:17:21,078 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:17:21,168 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:17:21,215 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:17:21,215 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:17:21,215 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:17:21,224 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318794 for request 300-0000331819 2018-11-29 21:17:21,231 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:17:21,231 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:17:21,232 INFO nodepool.NodeLauncher-0000318794: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318794 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318794 2018-11-29 21:17:21,233 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:17:22,173 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.9399137496948242s 2018-11-29 21:17:22,173 ERROR nodepool.NodeLauncher-0000318794: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318794: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:17:22,174 INFO nodepool.NodeLauncher-0000318794: Quota exceeded, invalidating quota cache 2018-11-29 21:17:22,174 INFO nodepool.NodeLauncher-0000318794: Aborting node 0000318794 due to quota failure 2018-11-29 21:17:23,993 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:17:23,995 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:17:23,997 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:17:23,997 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:17:23,998 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:17:23,998 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:17:23,999 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:17:24,000 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:17:24,000 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:17:24,000 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:17:24,000 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:17:24,001 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:17:24,001 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:17:24,002 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:17:24,307 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318793 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:17:24,307 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:17:24,310 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318793, state=deleting, external_id=None 2018-11-29 21:17:25,013 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:17:26,238 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:26,273 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2597167491912842s 2018-11-29 21:17:26,288 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:27,001 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7620174884796143s 2018-11-29 21:17:27,370 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.080035924911499s 2018-11-29 21:17:31,235 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:17:31,462 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.22655463218688965s 2018-11-29 21:17:31,463 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:17:31,463 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:17:31,588 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.12498044967651367s 2018-11-29 21:17:31,591 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:32,179 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5876359939575195s 2018-11-29 21:17:32,198 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:17:32,198 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:17:32,301 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:17:32,303 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:32,305 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:17:32,362 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:17:32,362 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:17:32,362 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:17:32,373 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318795 for request 300-0000331925 2018-11-29 21:17:32,380 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:17:32,380 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:17:32,382 INFO nodepool.NodeLauncher-0000318795: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318795 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318795 2018-11-29 21:17:32,382 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:17:33,131 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8277912139892578s 2018-11-29 21:17:33,635 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3340859413146973s 2018-11-29 21:17:33,651 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:34,633 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9800353050231934s 2018-11-29 21:17:34,735 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 2.35252046585083s 2018-11-29 21:17:34,735 ERROR nodepool.NodeLauncher-0000318795: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318795: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:17:34,736 INFO nodepool.NodeLauncher-0000318795: Quota exceeded, invalidating quota cache 2018-11-29 21:17:34,736 INFO nodepool.NodeLauncher-0000318795: Aborting node 0000318795 due to quota failure 2018-11-29 21:17:34,757 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:17:34,757 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:17:34,758 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:17:34,758 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:17:34,759 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:17:34,759 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:17:34,759 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:17:34,760 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:17:34,762 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:17:34,762 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:17:34,762 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:17:34,762 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:17:34,764 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:17:34,765 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:17:34,914 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318794 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:17:34,914 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:17:34,919 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318794, state=deleting, external_id=None 2018-11-29 21:17:39,026 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:39,573 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.5469837188720703s 2018-11-29 21:17:39,664 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:40,446 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.7818179130554199s 2018-11-29 21:17:40,459 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:17:41,777 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.31768798828125s 2018-11-29 21:17:42,384 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:17:42,573 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.18877387046813965s 2018-11-29 21:17:42,574 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:17:42,574 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:17:42,765 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.191392183303833s 2018-11-29 21:17:42,769 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:43,124 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.3548245429992676s 2018-11-29 21:17:43,144 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:17:43,144 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:17:43,260 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:17:43,312 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:17:43,312 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:17:43,313 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:17:43,325 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318796 for request 300-0000331819 2018-11-29 21:17:43,333 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:17:43,333 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:17:43,335 INFO nodepool.NodeLauncher-0000318796: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318796 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318796 2018-11-29 21:17:43,336 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:17:44,849 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.5128607749938965s 2018-11-29 21:17:44,849 ERROR nodepool.NodeLauncher-0000318796: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318796: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:17:44,860 INFO nodepool.NodeLauncher-0000318796: Quota exceeded, invalidating quota cache 2018-11-29 21:17:44,861 INFO nodepool.NodeLauncher-0000318796: Aborting node 0000318796 due to quota failure 2018-11-29 21:17:45,155 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:45,528 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:17:45,529 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:17:45,529 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:17:45,530 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:17:45,531 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:17:45,531 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:17:45,532 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:17:45,532 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:17:45,533 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:17:45,534 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:17:45,534 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:17:45,535 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:17:45,535 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:17:45,538 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:17:45,677 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318795 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:17:45,677 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:17:45,680 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318795, state=deleting, external_id=None 2018-11-29 21:17:45,846 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:45,988 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8326032161712646s 2018-11-29 21:17:46,614 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.7678687572479248s 2018-11-29 21:17:48,631 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:17:50,205 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.573777198791504s 2018-11-29 21:17:51,600 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:51,918 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:52,423 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8230397701263428s 2018-11-29 21:17:52,680 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.7618858814239502s 2018-11-29 21:17:53,338 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:17:53,544 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.20602130889892578s 2018-11-29 21:17:53,544 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:17:53,545 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:17:53,668 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.12339615821838379s 2018-11-29 21:17:53,672 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:54,271 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.59908127784729s 2018-11-29 21:17:54,289 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:17:54,290 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:17:54,390 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:17:54,436 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:17:54,436 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:17:54,436 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:17:54,447 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318797 for request 300-0000331925 2018-11-29 21:17:54,454 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:17:54,454 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:17:54,456 INFO nodepool.NodeLauncher-0000318797: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318797 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318797 2018-11-29 21:17:54,456 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:17:55,338 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.8818385601043701s 2018-11-29 21:17:55,339 ERROR nodepool.NodeLauncher-0000318797: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318797: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:17:55,340 INFO nodepool.NodeLauncher-0000318797: Quota exceeded, invalidating quota cache 2018-11-29 21:17:55,340 INFO nodepool.NodeLauncher-0000318797: Aborting node 0000318797 due to quota failure 2018-11-29 21:17:56,234 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:17:56,240 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:17:56,240 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:17:56,245 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:17:56,245 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:17:56,246 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:17:56,247 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:17:56,248 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:17:56,248 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:17:56,248 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:17:56,249 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:17:56,249 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:17:56,249 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:17:56,250 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:17:56,254 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:17:56,371 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318796 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:17:56,371 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:17:56,375 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318796, state=deleting, external_id=None 2018-11-29 21:17:57,677 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4427521228790283s 2018-11-29 21:17:58,021 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:58,366 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:17:58,633 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6114108562469482s 2018-11-29 21:17:58,842 ERROR nodepool.NodeDeleter: Exception deleting instance d075965c-a5c0-41e0-97bc-70050e66726e from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server d075965c-a5c0-41e0-97bc-70050e66726e deletion 2018-11-29 21:17:59,294 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.927983283996582s 2018-11-29 21:18:01,511 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211291 (state: deleting, allocated_to: None) 2018-11-29 21:18:01,511 INFO nodepool.DeletedNodeWorker: Deleting deleting instance d075965c-a5c0-41e0-97bc-70050e66726e from rdo-cloud-tripleo 2018-11-29 21:18:01,516 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server d075965c-a5c0-41e0-97bc-70050e66726e 2018-11-29 21:18:01,517 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:18:01,586 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.06866168975830078s 2018-11-29 21:18:03,708 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:18:04,447 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:04,459 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:18:04,661 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.20184826850891113s 2018-11-29 21:18:04,661 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:18:04,662 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:18:04,789 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.12752437591552734s 2018-11-29 21:18:04,793 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:05,040 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3315701484680176s 2018-11-29 21:18:05,056 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:05,626 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.8327956199645996s 2018-11-29 21:18:05,643 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:18:05,644 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:18:05,680 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.2335071563720703s 2018-11-29 21:18:05,750 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:18:05,795 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:18:05,795 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:18:05,795 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:18:05,804 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318798 for request 300-0000331819 2018-11-29 21:18:05,810 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:18:05,810 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:18:05,812 INFO nodepool.NodeLauncher-0000318798: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318798 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318798 2018-11-29 21:18:05,812 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:18:06,402 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.3423523902893066s 2018-11-29 21:18:07,116 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.3034050464630127s 2018-11-29 21:18:07,116 ERROR nodepool.NodeLauncher-0000318798: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318798: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:18:07,121 INFO nodepool.NodeLauncher-0000318798: Quota exceeded, invalidating quota cache 2018-11-29 21:18:07,121 INFO nodepool.NodeLauncher-0000318798: Aborting node 0000318798 due to quota failure 2018-11-29 21:18:07,219 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:18:07,219 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:18:07,220 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:18:07,221 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:18:07,222 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:18:07,222 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:18:07,223 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:18:07,226 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:18:07,226 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:18:07,226 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:18:07,227 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:18:07,227 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:18:07,227 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:18:07,228 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:18:07,312 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318797 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:18:07,313 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:18:07,318 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318797, state=deleting, external_id=None 2018-11-29 21:18:07,709 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:08,145 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.43535566329956055s 2018-11-29 21:18:08,145 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-cd60a7ae-803f-4f45-b5d6-7089d54c6832) 2018-11-29 21:18:08,276 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:18:11,071 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:18:11,694 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:12,347 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6527376174926758s 2018-11-29 21:18:12,481 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4099304676055908s 2018-11-29 21:18:12,496 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:13,356 ERROR nodepool.NodeDeleter: Exception deleting instance d0021aac-4e31-42c5-a6a6-b71097532a56 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server d0021aac-4e31-42c5-a6a6-b71097532a56 deletion 2018-11-29 21:18:14,053 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.5537762641906738s 2018-11-29 21:18:15,814 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:18:16,005 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.1900780200958252s 2018-11-29 21:18:16,005 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:18:16,006 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:18:16,142 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13611841201782227s 2018-11-29 21:18:16,145 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:16,675 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5289361476898193s 2018-11-29 21:18:16,693 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:18:16,693 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:18:16,796 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:18:16,845 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:18:16,845 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:18:16,845 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:18:16,857 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318799 for request 300-0000331925 2018-11-29 21:18:16,863 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:18:16,864 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:18:16,865 INFO nodepool.NodeLauncher-0000318799: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318799 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318799 2018-11-29 21:18:16,865 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:18:17,928 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:18:17,928 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:18:17,929 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:18:17,930 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:18:17,931 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:18:17,931 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:18:17,933 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:18:17,933 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:18:17,933 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:18:17,933 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:18:17,934 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:18:17,934 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:18:17,934 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:18:17,935 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:18:18,047 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318798 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:18:18,047 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:18:18,050 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318798, state=deleting, external_id=None 2018-11-29 21:18:18,125 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000242494 (state: deleting, allocated_to: None) 2018-11-29 21:18:18,125 INFO nodepool.DeletedNodeWorker: Deleting deleting instance d0021aac-4e31-42c5-a6a6-b71097532a56 from rdo-cloud-tripleo 2018-11-29 21:18:18,131 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server d0021aac-4e31-42c5-a6a6-b71097532a56 2018-11-29 21:18:18,131 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:18:18,199 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.06745433807373047s 2018-11-29 21:18:18,361 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:18,510 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:18:18,514 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.6488656997680664s 2018-11-29 21:18:18,515 ERROR nodepool.NodeLauncher-0000318799: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318799: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:18:18,515 INFO nodepool.NodeLauncher-0000318799: Quota exceeded, invalidating quota cache 2018-11-29 21:18:18,515 INFO nodepool.NodeLauncher-0000318799: Aborting node 0000318799 due to quota failure 2018-11-29 21:18:18,797 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.4354672431945801s 2018-11-29 21:18:19,745 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2344870567321777s 2018-11-29 21:18:19,761 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:21,066 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.3026304244995117s 2018-11-29 21:18:23,298 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318745 (state: used, allocated_to: 100-0000331948) 2018-11-29 21:18:23,299 INFO nodepool.DeletedNodeWorker: Deleting used instance None from ansible-statics 2018-11-29 21:18:23,302 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318745, state=deleting, external_id=None 2018-11-29 21:18:23,721 DEBUG nodepool.driver.static.StaticNodeProvider: Registered static node 35.236.232.200 2018-11-29 21:18:24,678 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:25,336 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6578223705291748s 2018-11-29 21:18:25,775 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:18:26,868 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:18:27,066 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2912242412567139s 2018-11-29 21:18:27,077 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2092912197113037s 2018-11-29 21:18:27,078 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:27,083 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:18:27,089 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:18:27,224 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13387227058410645s 2018-11-29 21:18:27,227 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:28,051 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.823868989944458s 2018-11-29 21:18:28,072 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:18:28,073 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:18:28,127 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0411381721496582s 2018-11-29 21:18:28,734 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:18:28,737 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:18:28,742 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:18:28,745 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:18:28,745 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:18:28,746 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:18:28,755 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:18:28,761 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:18:28,761 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:18:28,765 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:18:28,766 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:18:28,766 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:18:28,767 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:18:28,771 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:18:28,908 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:18:28,981 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:18:28,982 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:18:28,982 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:18:28,992 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318801 for request 300-0000331819 2018-11-29 21:18:28,999 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:18:28,999 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:18:29,001 INFO nodepool.NodeLauncher-0000318801: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318801 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318801 2018-11-29 21:18:29,001 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:18:30,434 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4326653480529785s 2018-11-29 21:18:30,435 ERROR nodepool.NodeLauncher-0000318801: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318801: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:18:30,435 INFO nodepool.NodeLauncher-0000318801: Quota exceeded, invalidating quota cache 2018-11-29 21:18:30,435 INFO nodepool.NodeLauncher-0000318801: Aborting node 0000318801 due to quota failure 2018-11-29 21:18:30,818 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:31,365 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.5472698211669922s 2018-11-29 21:18:33,095 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:18:34,096 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318799 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:18:34,097 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:18:34,102 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318799, state=deleting, external_id=None 2018-11-29 21:18:34,415 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3193168640136719s 2018-11-29 21:18:34,431 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:35,027 ERROR nodepool.NodeDeleter: Exception deleting instance 75e44b32-409c-43fd-9ec9-fa590b96b917 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 75e44b32-409c-43fd-9ec9-fa590b96b917 deletion 2018-11-29 21:18:35,650 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.219466209411621s 2018-11-29 21:18:37,362 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:37,666 ERROR nodepool.NodeDeleter: Exception deleting instance 33034fa9-59e6-48d7-88ee-cd4f0bb01e35 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 33034fa9-59e6-48d7-88ee-cd4f0bb01e35 deletion 2018-11-29 21:18:37,974 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6116189956665039s 2018-11-29 21:18:39,037 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:18:39,186 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.14855194091796875s 2018-11-29 21:18:39,186 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:18:39,187 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:18:39,376 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.18802690505981445s 2018-11-29 21:18:39,379 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:39,494 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:18:39,494 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:18:39,494 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:18:39,496 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:18:39,497 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:18:39,497 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:18:39,498 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:18:39,498 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:18:39,499 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:18:39,503 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:18:39,503 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:18:39,504 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:18:39,504 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:18:39,506 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:18:39,545 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211297 (state: deleting, allocated_to: None) 2018-11-29 21:18:39,545 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 75e44b32-409c-43fd-9ec9-fa590b96b917 from rdo-cloud-tripleo 2018-11-29 21:18:39,550 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 75e44b32-409c-43fd-9ec9-fa590b96b917 2018-11-29 21:18:39,550 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:18:39,566 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211288 (state: deleting, allocated_to: None) 2018-11-29 21:18:39,566 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 33034fa9-59e6-48d7-88ee-cd4f0bb01e35 from rdo-cloud-tripleo 2018-11-29 21:18:39,572 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 33034fa9-59e6-48d7-88ee-cd4f0bb01e35 2018-11-29 21:18:39,627 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.07615089416503906s 2018-11-29 21:18:39,629 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:18:39,699 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.06933093070983887s 2018-11-29 21:18:40,113 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.7330479621887207s 2018-11-29 21:18:40,131 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:18:40,132 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:18:40,260 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:18:40,330 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:18:40,330 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:18:40,330 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:18:40,343 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318802 for request 300-0000331925 2018-11-29 21:18:40,350 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:18:40,350 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:18:40,352 INFO nodepool.NodeLauncher-0000318802: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318802 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318802 2018-11-29 21:18:40,353 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:18:40,447 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:18:41,526 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.173029899597168s 2018-11-29 21:18:41,526 ERROR nodepool.NodeLauncher-0000318802: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318802: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:18:41,527 INFO nodepool.NodeLauncher-0000318802: Quota exceeded, invalidating quota cache 2018-11-29 21:18:41,527 INFO nodepool.NodeLauncher-0000318802: Aborting node 0000318802 due to quota failure 2018-11-29 21:18:41,744 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2966954708099365s 2018-11-29 21:18:41,759 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:42,738 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.978095293045044s 2018-11-29 21:18:43,392 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:44,184 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7924139499664307s 2018-11-29 21:18:44,711 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318801 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:18:44,712 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:18:44,715 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318801, state=deleting, external_id=None 2018-11-29 21:18:46,168 ERROR nodepool.NodeDeleter: Exception deleting instance 6a0c9d49-2789-4188-92f2-aa94b2491ce5 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 6a0c9d49-2789-4188-92f2-aa94b2491ce5 deletion 2018-11-29 21:18:46,756 ERROR nodepool.NodeDeleter: Exception deleting instance 3f4f9477-ce7a-4bd1-af21-8ce7eedc74e8 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 3f4f9477-ce7a-4bd1-af21-8ce7eedc74e8 deletion 2018-11-29 21:18:47,775 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:49,601 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.826019525527954s 2018-11-29 21:18:49,616 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:18:49,996 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:50,264 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:18:50,265 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:18:50,266 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:18:50,266 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:18:50,268 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:18:50,268 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:18:50,269 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:18:50,269 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:18:50,270 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:18:50,271 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:18:50,271 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:18:50,273 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:18:50,274 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:18:50,274 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:18:50,274 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211294 (state: deleting, allocated_to: None) 2018-11-29 21:18:50,275 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 6a0c9d49-2789-4188-92f2-aa94b2491ce5 from rdo-cloud-tripleo 2018-11-29 21:18:50,281 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 6a0c9d49-2789-4188-92f2-aa94b2491ce5 2018-11-29 21:18:50,321 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211289 (state: deleting, allocated_to: None) 2018-11-29 21:18:50,321 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 3f4f9477-ce7a-4bd1-af21-8ce7eedc74e8 from rdo-cloud-tripleo 2018-11-29 21:18:50,325 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 3f4f9477-ce7a-4bd1-af21-8ce7eedc74e8 2018-11-29 21:18:50,362 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:18:50,541 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.17811942100524902s 2018-11-29 21:18:50,541 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:18:50,541 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:18:50,612 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6161115169525146s 2018-11-29 21:18:50,670 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.12808585166931152s 2018-11-29 21:18:50,673 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:50,920 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3034899234771729s 2018-11-29 21:18:50,936 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 1) 2018-11-29 21:18:51,007 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.07113122940063477s 2018-11-29 21:18:51,009 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:18:51,077 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.0670921802520752s 2018-11-29 21:18:51,278 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6042497158050537s 2018-11-29 21:18:51,295 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:18:51,295 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:18:51,404 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:18:51,456 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:18:51,456 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:18:51,456 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:18:51,465 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318803 for request 300-0000331819 2018-11-29 21:18:51,476 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:18:51,476 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:18:51,478 INFO nodepool.NodeLauncher-0000318803: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318803 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318803 2018-11-29 21:18:51,478 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:18:52,401 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.922421932220459s 2018-11-29 21:18:52,401 ERROR nodepool.NodeLauncher-0000318803: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318803: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:18:52,402 INFO nodepool.NodeLauncher-0000318803: Quota exceeded, invalidating quota cache 2018-11-29 21:18:52,402 INFO nodepool.NodeLauncher-0000318803: Aborting node 0000318803 due to quota failure 2018-11-29 21:18:52,945 ERROR nodepool.NodeDeleter: Exception deleting instance b4853e6a-3115-451a-96cb-3c751068e393 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server b4853e6a-3115-451a-96cb-3c751068e393 deletion 2018-11-29 21:18:54,807 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:55,468 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318802 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:18:55,468 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:18:55,471 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318802, state=deleting, external_id=None 2018-11-29 21:18:55,570 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211299 (state: deleting, allocated_to: None) 2018-11-29 21:18:55,570 INFO nodepool.DeletedNodeWorker: Deleting deleting instance b4853e6a-3115-451a-96cb-3c751068e393 from rdo-cloud-tripleo 2018-11-29 21:18:55,576 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server b4853e6a-3115-451a-96cb-3c751068e393 2018-11-29 21:18:56,006 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1982977390289307s 2018-11-29 21:18:56,019 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:18:56,097 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.07741594314575195s 2018-11-29 21:18:56,208 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:18:56,951 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:18:57,053 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8446817398071289s 2018-11-29 21:18:58,103 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1516592502593994s 2018-11-29 21:19:01,025 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:01,067 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:19:01,069 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:19:01,070 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:19:01,070 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:19:01,070 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:19:01,071 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:19:01,072 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:19:01,075 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:19:01,075 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:19:01,076 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:19:01,076 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:19:01,076 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:19:01,076 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:19:01,076 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:19:01,482 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:19:01,734 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2515134811401367s 2018-11-29 21:19:01,734 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:19:01,735 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:19:01,933 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.19846796989440918s 2018-11-29 21:19:01,937 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:02,255 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2302188873291016s 2018-11-29 21:19:02,337 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.3996438980102539s 2018-11-29 21:19:02,355 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:19:02,356 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:19:02,473 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:19:02,534 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:19:02,534 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:19:02,534 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:19:02,544 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318804 for request 300-0000331925 2018-11-29 21:19:02,551 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:19:02,551 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:19:02,553 INFO nodepool.NodeLauncher-0000318804: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318804 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318804 2018-11-29 21:19:02,554 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:19:02,633 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:03,652 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.0184001922607422s 2018-11-29 21:19:04,099 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.5450501441955566s 2018-11-29 21:19:04,099 ERROR nodepool.NodeLauncher-0000318804: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318804: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:19:04,100 INFO nodepool.NodeLauncher-0000318804: Quota exceeded, invalidating quota cache 2018-11-29 21:19:04,100 INFO nodepool.NodeLauncher-0000318804: Aborting node 0000318804 due to quota failure 2018-11-29 21:19:04,133 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:19:05,461 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.327458143234253s 2018-11-29 21:19:06,346 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318803 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:19:06,346 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:19:06,353 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318803, state=deleting, external_id=None 2018-11-29 21:19:07,439 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:08,187 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.7476751804351807s 2018-11-29 21:19:08,676 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:09,074 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.39760923385620117s 2018-11-29 21:19:09,074 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-055fa1d3-4b0f-4662-afd0-830f138c3d80) 2018-11-29 21:19:09,076 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:09,201 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:19:10,159 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.0827815532684326s 2018-11-29 21:19:11,492 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:19:11,772 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:19:11,774 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:19:11,774 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:19:11,775 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:19:11,777 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:19:11,777 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:19:11,778 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:19:11,778 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:19:11,779 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:19:11,780 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:19:11,780 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:19:11,781 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:19:11,781 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:19:11,782 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:19:12,051 ERROR nodepool.NodeDeleter: Exception deleting instance 9c02645e-baeb-4e9e-8a50-62d4671f8c9c from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 9c02645e-baeb-4e9e-8a50-62d4671f8c9c deletion 2018-11-29 21:19:12,556 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:19:12,809 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2527906894683838s 2018-11-29 21:19:12,809 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:19:12,810 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:19:13,054 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.24378585815429688s 2018-11-29 21:19:13,057 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:13,089 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.5970172882080078s 2018-11-29 21:19:13,571 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:14,192 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 1.1342897415161133s 2018-11-29 21:19:14,211 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:19:14,213 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:19:14,341 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:19:14,403 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:19:14,403 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:19:14,404 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:19:14,417 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318805 for request 300-0000331819 2018-11-29 21:19:14,425 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:19:14,425 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:19:14,427 INFO nodepool.NodeLauncher-0000318805: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318805 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318805 2018-11-29 21:19:14,428 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:19:14,559 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9872970581054688s 2018-11-29 21:19:15,680 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:16,299 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6186327934265137s 2018-11-29 21:19:16,426 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.9983768463134766s 2018-11-29 21:19:16,427 ERROR nodepool.NodeLauncher-0000318805: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318805: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:19:16,427 INFO nodepool.NodeLauncher-0000318805: Quota exceeded, invalidating quota cache 2018-11-29 21:19:16,427 INFO nodepool.NodeLauncher-0000318805: Aborting node 0000318805 due to quota failure 2018-11-29 21:19:17,003 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318804 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:19:17,003 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:19:17,006 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318804, state=deleting, external_id=None 2018-11-29 21:19:17,222 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000242496 (state: deleting, allocated_to: None) 2018-11-29 21:19:17,222 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 9c02645e-baeb-4e9e-8a50-62d4671f8c9c from rdo-cloud-tripleo 2018-11-29 21:19:17,227 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 9c02645e-baeb-4e9e-8a50-62d4671f8c9c 2018-11-29 21:19:17,228 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:19:17,296 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.06811118125915527s 2018-11-29 21:19:19,118 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:19:20,393 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2749600410461426s 2018-11-29 21:19:20,410 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:21,482 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0717682838439941s 2018-11-29 21:19:22,183 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:22,547 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:19:22,549 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:19:22,551 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:19:22,551 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:19:22,552 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:19:22,552 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:19:22,555 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:19:22,555 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:19:22,555 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:19:22,556 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:19:22,557 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:19:22,557 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:19:22,558 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:19:22,559 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:19:22,803 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6200871467590332s 2018-11-29 21:19:24,431 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:19:24,740 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.3090341091156006s 2018-11-29 21:19:24,740 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:19:24,741 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:19:24,889 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.14835786819458008s 2018-11-29 21:19:24,893 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:25,190 ERROR nodepool.NodeDeleter: Exception deleting instance 0f13ac4c-c10c-4892-8172-561e6f949d19 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 0f13ac4c-c10c-4892-8172-561e6f949d19 deletion 2018-11-29 21:19:25,907 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 1.0139827728271484s 2018-11-29 21:19:25,925 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:19:25,925 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:19:26,052 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:19:26,106 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:19:26,106 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:19:26,106 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:19:26,115 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318806 for request 300-0000331925 2018-11-29 21:19:26,124 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:19:26,124 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:19:26,126 INFO nodepool.NodeLauncher-0000318806: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318806 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318806 2018-11-29 21:19:26,126 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:19:26,424 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:19:27,475 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.3479082584381104s 2018-11-29 21:19:27,475 ERROR nodepool.NodeLauncher-0000318806: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318806: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:19:27,475 INFO nodepool.NodeLauncher-0000318806: Quota exceeded, invalidating quota cache 2018-11-29 21:19:27,476 INFO nodepool.NodeLauncher-0000318806: Aborting node 0000318806 due to quota failure 2018-11-29 21:19:27,739 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3148880004882812s 2018-11-29 21:19:27,745 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:27,765 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318805 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:19:27,766 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:19:27,769 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318805, state=deleting, external_id=None 2018-11-29 21:19:27,820 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211293 (state: deleting, allocated_to: None) 2018-11-29 21:19:27,821 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 0f13ac4c-c10c-4892-8172-561e6f949d19 from rdo-cloud-tripleo 2018-11-29 21:19:27,828 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 0f13ac4c-c10c-4892-8172-561e6f949d19 2018-11-29 21:19:28,323 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:28,533 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.7832343578338623s 2018-11-29 21:19:28,547 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:19:28,613 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.06613516807556152s 2018-11-29 21:19:29,014 ERROR nodepool.NodeDeleter: Exception deleting instance 52af0012-6653-4e5d-adcf-291841fe4c5f from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server 52af0012-6653-4e5d-adcf-291841fe4c5f deletion 2018-11-29 21:19:29,377 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.0531830787658691s 2018-11-29 21:19:33,308 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:19:33,310 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:19:33,313 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:19:33,314 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:19:33,315 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:19:33,318 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:19:33,318 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:19:33,319 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:19:33,320 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:19:33,320 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:19:33,320 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:19:33,321 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:19:33,321 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:19:33,322 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:19:33,340 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211295 (state: deleting, allocated_to: None) 2018-11-29 21:19:33,341 INFO nodepool.DeletedNodeWorker: Deleting deleting instance 52af0012-6653-4e5d-adcf-291841fe4c5f from rdo-cloud-tripleo 2018-11-29 21:19:33,348 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server 52af0012-6653-4e5d-adcf-291841fe4c5f 2018-11-29 21:19:33,349 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:19:33,432 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.08347368240356445s 2018-11-29 21:19:33,746 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:34,828 ERROR nodepool.NodeDeleter: Exception deleting instance e5999063-a1b8-412a-bd0e-ef26c58f0381 from rdo-cloud: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server e5999063-a1b8-412a-bd0e-ef26c58f0381 deletion 2018-11-29 21:19:35,139 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.392430305480957s 2018-11-29 21:19:35,152 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:19:35,392 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:36,129 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:19:36,201 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8089675903320312s 2018-11-29 21:19:36,274 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.14426517486572266s 2018-11-29 21:19:36,274 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:19:36,275 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:19:36,408 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1332228183746338s 2018-11-29 21:19:36,411 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:36,482 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3298084735870361s 2018-11-29 21:19:37,625 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 1.213714838027954s 2018-11-29 21:19:37,643 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:19:37,643 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:19:37,756 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:19:37,815 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:19:37,815 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:19:37,815 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:19:37,826 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318807 for request 300-0000331819 2018-11-29 21:19:37,834 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:19:37,834 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:19:37,836 INFO nodepool.NodeLauncher-0000318807: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318807 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318807 2018-11-29 21:19:37,836 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:19:38,527 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318806 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:19:38,527 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:19:38,534 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318806, state=deleting, external_id=None 2018-11-29 21:19:38,890 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.0537669658660889s 2018-11-29 21:19:38,891 ERROR nodepool.NodeLauncher-0000318807: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318807: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:19:38,891 INFO nodepool.NodeLauncher-0000318807: Quota exceeded, invalidating quota cache 2018-11-29 21:19:38,891 INFO nodepool.NodeLauncher-0000318807: Aborting node 0000318807 due to quota failure 2018-11-29 21:19:38,960 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000260472 (state: deleting, allocated_to: None) 2018-11-29 21:19:38,961 INFO nodepool.DeletedNodeWorker: Deleting deleting instance e5999063-a1b8-412a-bd0e-ef26c58f0381 from rdo-cloud 2018-11-29 21:19:38,966 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server e5999063-a1b8-412a-bd0e-ef26c58f0381 2018-11-29 21:19:38,967 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeDeleteServers (queue 0) 2018-11-29 21:19:39,042 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeDeleteServers in 0.07522845268249512s 2018-11-29 21:19:40,170 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:41,589 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.4190480709075928s 2018-11-29 21:19:42,215 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:42,513 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:19:42,954 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7383286952972412s 2018-11-29 21:19:44,022 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.5085430145263672s 2018-11-29 21:19:44,041 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:19:44,043 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:19:44,044 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:19:44,044 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:19:44,045 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:19:44,045 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:19:44,046 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:19:44,046 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:19:44,046 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:19:44,046 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:19:44,047 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:19:44,047 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:19:44,048 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:19:44,048 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:19:46,601 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:47,451 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8494577407836914s 2018-11-29 21:19:47,840 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:19:48,267 ERROR nodepool.NodeDeleter: Exception deleting instance bbbdbcbc-cfa9-4ee1-828e-03a90bdda5ac from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server bbbdbcbc-cfa9-4ee1-828e-03a90bdda5ac deletion 2018-11-29 21:19:48,281 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.4410884380340576s 2018-11-29 21:19:48,281 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:19:48,282 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:19:48,415 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13274645805358887s 2018-11-29 21:19:48,418 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:48,969 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:49,125 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.7074310779571533s 2018-11-29 21:19:49,144 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:19:49,144 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:19:49,254 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:19:49,345 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:19:49,345 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:19:49,345 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:19:49,358 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318808 for request 300-0000331925 2018-11-29 21:19:49,368 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:19:49,369 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:19:49,372 INFO nodepool.NodeLauncher-0000318808: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318808 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318808 2018-11-29 21:19:49,373 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:19:49,382 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.4128594398498535s 2018-11-29 21:19:49,476 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000242421 (state: deleting, allocated_to: None) 2018-11-29 21:19:49,476 INFO nodepool.DeletedNodeWorker: Deleting deleting instance bbbdbcbc-cfa9-4ee1-828e-03a90bdda5ac from rdo-cloud-tripleo 2018-11-29 21:19:49,483 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server bbbdbcbc-cfa9-4ee1-828e-03a90bdda5ac 2018-11-29 21:19:49,484 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:19:49,551 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.06711149215698242s 2018-11-29 21:19:50,052 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:19:50,857 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4843425750732422s 2018-11-29 21:19:50,858 ERROR nodepool.NodeLauncher-0000318808: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318808: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:19:50,859 INFO nodepool.NodeLauncher-0000318808: Quota exceeded, invalidating quota cache 2018-11-29 21:19:50,859 INFO nodepool.NodeLauncher-0000318808: Aborting node 0000318808 due to quota failure 2018-11-29 21:19:51,262 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2088673114776611s 2018-11-29 21:19:52,640 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:53,505 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8643701076507568s 2018-11-29 21:19:54,653 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318807 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:19:54,653 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:19:54,662 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318807, state=deleting, external_id=None 2018-11-29 21:19:54,758 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:19:54,764 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:19:54,764 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:19:54,766 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:19:54,766 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:19:54,767 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:19:54,767 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:19:54,767 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:19:54,768 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:19:54,768 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:19:54,769 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:19:54,769 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:19:54,769 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:19:54,770 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:19:55,068 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:56,345 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.27677321434021s 2018-11-29 21:19:57,289 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:19:58,480 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1904311180114746s 2018-11-29 21:19:58,667 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:19:59,373 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:19:59,606 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.23292803764343262s 2018-11-29 21:19:59,606 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:19:59,607 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:19:59,630 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9628434181213379s 2018-11-29 21:19:59,745 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13798809051513672s 2018-11-29 21:19:59,748 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:00,228 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.480165958404541s 2018-11-29 21:20:00,246 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:20:00,247 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:20:00,387 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:20:00,456 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:20:00,456 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:20:00,457 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:20:00,467 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318809 for request 300-0000331819 2018-11-29 21:20:00,474 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:20:00,474 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:20:00,476 INFO nodepool.NodeLauncher-0000318809: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318809 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318809 2018-11-29 21:20:00,476 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:20:01,392 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 0.9155149459838867s 2018-11-29 21:20:01,392 ERROR nodepool.NodeLauncher-0000318809: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318809: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:20:01,393 INFO nodepool.NodeLauncher-0000318809: Quota exceeded, invalidating quota cache 2018-11-29 21:20:01,393 INFO nodepool.NodeLauncher-0000318809: Aborting node 0000318809 due to quota failure 2018-11-29 21:20:01,405 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:01,857 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.45172595977783203s 2018-11-29 21:20:04,592 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:20:05,366 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318808 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:20:05,366 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:20:05,373 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318808, state=deleting, external_id=None 2018-11-29 21:20:05,507 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:20:05,510 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:20:05,510 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:20:05,511 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:20:05,511 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:20:05,513 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:20:05,513 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:20:05,513 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:20:05,513 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:20:05,515 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:20:05,515 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:20:05,515 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:20:05,516 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:20:05,516 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:20:05,966 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3743195533752441s 2018-11-29 21:20:05,982 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:07,147 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1612627506256104s 2018-11-29 21:20:07,873 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:08,496 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6225919723510742s 2018-11-29 21:20:09,666 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:10,010 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.3436157703399658s 2018-11-29 21:20:10,011 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-c92d8853-418b-46ec-90bc-e240bec0fd2e) 2018-11-29 21:20:10,148 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:20:10,480 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:20:10,769 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.28918910026550293s 2018-11-29 21:20:10,770 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:20:10,771 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:20:10,975 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.2042865753173828s 2018-11-29 21:20:10,979 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:11,740 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.7609617710113525s 2018-11-29 21:20:11,758 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:20:11,758 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:20:11,869 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:20:11,919 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:20:11,919 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:20:11,919 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:20:11,936 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318810 for request 300-0000331925 2018-11-29 21:20:11,944 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:20:11,945 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:20:11,946 INFO nodepool.NodeLauncher-0000318810: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318810 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318810 2018-11-29 21:20:11,947 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:20:11,997 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:20:13,458 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.4604182243347168s 2018-11-29 21:20:13,474 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:13,517 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.5705316066741943s 2018-11-29 21:20:13,518 ERROR nodepool.NodeLauncher-0000318810: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318810: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:20:13,518 INFO nodepool.NodeLauncher-0000318810: Quota exceeded, invalidating quota cache 2018-11-29 21:20:13,518 INFO nodepool.NodeLauncher-0000318810: Aborting node 0000318810 due to quota failure 2018-11-29 21:20:14,382 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:14,517 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.041154146194458s 2018-11-29 21:20:14,943 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.5601906776428223s 2018-11-29 21:20:16,109 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318809 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:20:16,110 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:20:16,119 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318809, state=deleting, external_id=None 2018-11-29 21:20:16,280 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:20:16,280 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:20:16,282 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:20:16,284 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:20:16,284 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:20:16,286 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:20:16,286 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:20:16,287 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:20:16,288 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:20:16,288 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:20:16,289 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:20:16,289 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:20:16,290 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:20:16,292 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:20:19,493 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:20:20,517 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:20,823 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3301050662994385s 2018-11-29 21:20:20,838 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:21,127 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6097466945648193s 2018-11-29 21:20:21,950 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:20:22,113 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.272261381149292s 2018-11-29 21:20:22,204 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2544081211090088s 2018-11-29 21:20:22,205 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:20:22,205 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:20:22,337 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13123512268066406s 2018-11-29 21:20:22,340 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:22,974 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6338346004486084s 2018-11-29 21:20:22,993 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:20:22,993 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:20:23,101 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:20:23,153 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:20:23,153 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:20:23,153 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:20:23,162 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318811 for request 300-0000331819 2018-11-29 21:20:23,171 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:20:23,171 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:20:23,173 INFO nodepool.NodeLauncher-0000318811: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318811 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318811 2018-11-29 21:20:23,173 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:20:24,613 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4398856163024902s 2018-11-29 21:20:24,613 ERROR nodepool.NodeLauncher-0000318811: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318811: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:20:24,614 INFO nodepool.NodeLauncher-0000318811: Quota exceeded, invalidating quota cache 2018-11-29 21:20:24,614 INFO nodepool.NodeLauncher-0000318811: Aborting node 0000318811 due to quota failure 2018-11-29 21:20:26,850 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:20:26,892 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318810 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:20:26,892 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:20:26,900 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318810, state=deleting, external_id=None 2018-11-29 21:20:26,966 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:27,011 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:20:27,011 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:20:27,015 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:20:27,016 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:20:27,016 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:20:27,018 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:20:27,018 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:20:27,018 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:20:27,019 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:20:27,019 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:20:27,020 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:20:27,021 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:20:27,022 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:20:27,022 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:20:27,556 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.5902690887451172s 2018-11-29 21:20:28,425 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.575392484664917s 2018-11-29 21:20:28,439 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:29,905 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.460886001586914s 2018-11-29 21:20:33,152 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:33,177 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:20:33,322 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.14492058753967285s 2018-11-29 21:20:33,323 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:20:33,323 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:20:33,518 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.19503283500671387s 2018-11-29 21:20:33,522 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:34,093 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9408962726593018s 2018-11-29 21:20:34,107 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5845859050750732s 2018-11-29 21:20:34,125 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:20:34,125 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:20:34,237 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:20:34,289 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:20:34,289 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:20:34,289 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:20:34,299 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318812 for request 300-0000331925 2018-11-29 21:20:34,306 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:20:34,306 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:20:34,308 INFO nodepool.NodeLauncher-0000318812: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318812 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318812 2018-11-29 21:20:34,308 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:20:34,455 ERROR nodepool.NodeDeleter: Exception deleting instance fbf7b562-5826-4b67-a10b-87b962a17f12 from rdo-cloud-tripleo: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 81, in delete manager.waitForNodeCleanup(node.external_id) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 361, in waitForNodeCleanup "server %s deletion" % server_id): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/nodeutils.py", line 41, in iterate_timeout raise exc("Timeout waiting for %s" % purpose) nodepool.exceptions.ServerDeleteException: Timeout waiting for server fbf7b562-5826-4b67-a10b-87b962a17f12 deletion 2018-11-29 21:20:35,249 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:35,661 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.3528404235839844s 2018-11-29 21:20:35,662 ERROR nodepool.NodeLauncher-0000318812: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318812: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:20:35,662 INFO nodepool.NodeLauncher-0000318812: Quota exceeded, invalidating quota cache 2018-11-29 21:20:35,662 INFO nodepool.NodeLauncher-0000318812: Aborting node 0000318812 due to quota failure 2018-11-29 21:20:36,757 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.5077176094055176s 2018-11-29 21:20:37,602 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318811 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:20:37,602 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:20:37,612 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318811, state=deleting, external_id=None 2018-11-29 21:20:37,765 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:20:37,767 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:20:37,767 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:20:37,768 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:20:37,768 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:20:37,769 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:20:37,769 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:20:37,771 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:20:37,771 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:20:37,772 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:20:37,772 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:20:37,773 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:20:37,773 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:20:37,777 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:20:37,812 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000211298 (state: deleting, allocated_to: None) 2018-11-29 21:20:37,813 INFO nodepool.DeletedNodeWorker: Deleting deleting instance fbf7b562-5826-4b67-a10b-87b962a17f12 from rdo-cloud-tripleo 2018-11-29 21:20:37,819 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:20:39,207 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3880581855773926s 2018-11-29 21:20:39,225 DEBUG nodepool.driver.openstack.OpenStackProvider: Deleting server fbf7b562-5826-4b67-a10b-87b962a17f12 2018-11-29 21:20:39,226 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeDeleteServers (queue 0) 2018-11-29 21:20:39,331 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeDeleteServers in 0.10507559776306152s 2018-11-29 21:20:39,581 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:40,001 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.41954803466796875s 2018-11-29 21:20:41,915 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:43,081 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1659862995147705s 2018-11-29 21:20:44,311 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:20:44,525 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.21336913108825684s 2018-11-29 21:20:44,525 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:20:44,525 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:20:44,721 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.19584298133850098s 2018-11-29 21:20:44,725 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:45,345 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:20:45,411 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6855998039245605s 2018-11-29 21:20:45,429 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:20:45,429 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:20:45,529 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:20:45,581 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:20:45,581 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:20:45,581 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:20:45,591 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318813 for request 300-0000331819 2018-11-29 21:20:45,600 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:20:45,600 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:20:45,601 INFO nodepool.NodeLauncher-0000318813: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318813 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318813 2018-11-29 21:20:45,602 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:20:46,029 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:46,616 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.5867977142333984s 2018-11-29 21:20:46,658 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3135523796081543s 2018-11-29 21:20:47,087 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4845521450042725s 2018-11-29 21:20:47,087 ERROR nodepool.NodeLauncher-0000318813: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318813: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:20:47,087 INFO nodepool.NodeLauncher-0000318813: Quota exceeded, invalidating quota cache 2018-11-29 21:20:47,088 INFO nodepool.NodeLauncher-0000318813: Aborting node 0000318813 due to quota failure 2018-11-29 21:20:48,168 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:48,355 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318812 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:20:48,355 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:20:48,363 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318812, state=deleting, external_id=None 2018-11-29 21:20:48,457 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:20:48,458 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:20:48,459 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:20:48,460 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:20:48,461 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:20:48,462 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:20:48,463 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:20:48,463 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:20:48,464 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:20:48,464 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:20:48,465 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:20:48,465 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:20:48,467 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:20:48,467 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:20:49,782 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.6132755279541016s 2018-11-29 21:20:52,129 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:52,686 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:20:53,146 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.016970157623291s 2018-11-29 21:20:54,073 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3871984481811523s 2018-11-29 21:20:54,800 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:55,605 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:20:55,812 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.20688819885253906s 2018-11-29 21:20:55,812 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:20:55,812 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:20:56,005 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.19252228736877441s 2018-11-29 21:20:56,009 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:56,034 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.2336044311523438s 2018-11-29 21:20:56,662 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6537113189697266s 2018-11-29 21:20:56,680 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:20:56,680 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:20:56,778 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:20:56,825 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:20:56,825 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:20:56,825 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:20:56,834 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318814 for request 300-0000331925 2018-11-29 21:20:56,841 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:20:56,841 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:20:56,843 INFO nodepool.NodeLauncher-0000318814: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318814 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318814 2018-11-29 21:20:56,843 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:20:58,052 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2088642120361328s 2018-11-29 21:20:58,052 ERROR nodepool.NodeLauncher-0000318814: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318814: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:20:58,053 INFO nodepool.NodeLauncher-0000318814: Quota exceeded, invalidating quota cache 2018-11-29 21:20:58,053 INFO nodepool.NodeLauncher-0000318814: Aborting node 0000318814 due to quota failure 2018-11-29 21:20:58,642 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:20:59,080 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318813 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:20:59,080 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:20:59,091 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318813, state=deleting, external_id=None 2018-11-29 21:20:59,195 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:20:59,197 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:20:59,199 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:20:59,200 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:20:59,200 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:20:59,200 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:20:59,201 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:20:59,201 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:20:59,201 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:20:59,202 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:20:59,202 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:20:59,202 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:20:59,203 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:20:59,204 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:20:59,452 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8102116584777832s 2018-11-29 21:21:00,217 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:21:01,731 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.51416015625s 2018-11-29 21:21:01,747 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:02,929 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.17734694480896s 2018-11-29 21:21:05,170 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:06,846 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:21:06,850 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.6794779300689697s 2018-11-29 21:21:07,083 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.23722386360168457s 2018-11-29 21:21:07,084 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:21:07,084 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:21:07,231 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.14602065086364746s 2018-11-29 21:21:07,236 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:07,762 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:21:07,834 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5977745056152344s 2018-11-29 21:21:07,853 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:21:07,853 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:21:07,962 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:21:08,012 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:21:08,012 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:21:08,012 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:21:08,022 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318815 for request 300-0000331819 2018-11-29 21:21:08,031 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:21:08,032 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:21:08,033 INFO nodepool.NodeLauncher-0000318815: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318815 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318815 2018-11-29 21:21:08,033 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:21:09,146 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.382126808166504s 2018-11-29 21:21:09,162 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:09,408 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.373999834060669s 2018-11-29 21:21:09,408 ERROR nodepool.NodeLauncher-0000318815: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318815: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:21:09,408 INFO nodepool.NodeLauncher-0000318815: Quota exceeded, invalidating quota cache 2018-11-29 21:21:09,408 INFO nodepool.NodeLauncher-0000318815: Aborting node 0000318815 due to quota failure 2018-11-29 21:21:09,713 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318814 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:21:09,713 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:21:09,721 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318814, state=deleting, external_id=None 2018-11-29 21:21:10,044 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:21:10,046 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:21:10,046 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:21:10,048 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:21:10,049 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:21:10,049 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:21:10,050 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:21:10,051 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:21:10,051 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:21:10,051 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:21:10,052 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:21:10,052 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:21:10,058 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:21:10,059 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:21:10,269 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1043570041656494s 2018-11-29 21:21:10,565 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:11,026 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.46118807792663574s 2018-11-29 21:21:11,027 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-8c5ff190-ecaf-4a3f-a40e-1227ef46c470) 2018-11-29 21:21:11,143 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:21:12,866 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:13,513 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6471495628356934s 2018-11-29 21:21:15,176 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:21:16,471 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.294823408126831s 2018-11-29 21:21:16,487 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:17,310 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8213636875152588s 2018-11-29 21:21:18,037 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:21:18,248 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2112278938293457s 2018-11-29 21:21:18,249 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:21:18,249 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:21:18,383 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13291335105895996s 2018-11-29 21:21:18,386 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:19,077 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6914446353912354s 2018-11-29 21:21:19,096 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:21:19,096 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:21:19,196 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:21:19,245 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:21:19,245 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:21:19,245 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:21:19,256 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318816 for request 300-0000331925 2018-11-29 21:21:19,263 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:21:19,263 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:21:19,265 INFO nodepool.NodeLauncher-0000318816: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318816 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318816 2018-11-29 21:21:19,266 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:21:19,491 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:20,133 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6422851085662842s 2018-11-29 21:21:20,579 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318815 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:21:20,580 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:21:20,590 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318815, state=deleting, external_id=None 2018-11-29 21:21:20,678 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.411952257156372s 2018-11-29 21:21:20,678 ERROR nodepool.NodeLauncher-0000318816: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318816: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:21:20,679 INFO nodepool.NodeLauncher-0000318816: Quota exceeded, invalidating quota cache 2018-11-29 21:21:20,680 INFO nodepool.NodeLauncher-0000318816: Aborting node 0000318816 due to quota failure 2018-11-29 21:21:20,836 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:21:20,837 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:21:20,838 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:21:20,839 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:21:20,839 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:21:20,841 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:21:20,841 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:21:20,841 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:21:20,845 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:21:20,846 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:21:20,846 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:21:20,846 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:21:20,847 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:21:20,851 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:21:22,497 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:23,269 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.7713885307312012s 2018-11-29 21:21:23,283 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:21:24,555 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2718582153320312s 2018-11-29 21:21:25,537 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:25,950 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.41342687606811523s 2018-11-29 21:21:28,317 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:29,268 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:21:29,333 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0156052112579346s 2018-11-29 21:21:29,455 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.18742036819458008s 2018-11-29 21:21:29,456 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:21:29,456 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:21:29,599 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.14299917221069336s 2018-11-29 21:21:29,603 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:29,996 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.393247127532959s 2018-11-29 21:21:30,020 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:21:30,020 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:21:30,123 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:21:30,175 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:21:30,175 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:21:30,175 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:21:30,189 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318817 for request 300-0000331819 2018-11-29 21:21:30,196 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:21:30,196 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:21:30,198 INFO nodepool.NodeLauncher-0000318817: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318817 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318817 2018-11-29 21:21:30,199 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:21:30,586 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:21:31,402 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318816 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:21:31,402 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:21:31,410 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318816, state=deleting, external_id=None 2018-11-29 21:21:31,562 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:21:31,564 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:21:31,564 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:21:31,566 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:21:31,566 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:21:31,567 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:21:31,567 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:21:31,568 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:21:31,568 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:21:31,569 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:21:31,570 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:21:31,570 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:21:31,570 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:21:31,571 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:21:31,751 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.5526461601257324s 2018-11-29 21:21:31,752 ERROR nodepool.NodeLauncher-0000318817: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318817: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:21:31,752 INFO nodepool.NodeLauncher-0000318817: Quota exceeded, invalidating quota cache 2018-11-29 21:21:31,752 INFO nodepool.NodeLauncher-0000318817: Aborting node 0000318817 due to quota failure 2018-11-29 21:21:31,781 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1945922374725342s 2018-11-29 21:21:31,963 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:33,084 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.120415449142456s 2018-11-29 21:21:34,553 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:35,839 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.285801887512207s 2018-11-29 21:21:37,811 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:21:38,170 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:38,853 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.683241605758667s 2018-11-29 21:21:39,131 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3201539516448975s 2018-11-29 21:21:40,202 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:21:40,422 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.21903038024902344s 2018-11-29 21:21:40,422 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:21:40,423 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:21:40,623 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.19976019859313965s 2018-11-29 21:21:40,626 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:41,008 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:41,440 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.8137860298156738s 2018-11-29 21:21:41,459 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:21:41,459 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:21:42,063 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:21:42,194 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1848278045654297s 2018-11-29 21:21:42,330 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:21:42,338 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:21:42,339 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:21:42,339 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:21:42,340 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:21:42,340 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:21:42,341 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:21:42,342 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:21:42,342 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:21:42,342 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:21:42,343 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:21:42,343 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:21:42,343 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:21:42,344 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:21:42,452 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:21:42,452 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:21:42,452 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:21:42,465 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318818 for request 300-0000331925 2018-11-29 21:21:42,476 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:21:42,476 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:21:42,478 INFO nodepool.NodeLauncher-0000318818: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318818 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318818 2018-11-29 21:21:42,479 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:21:43,520 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.0407190322875977s 2018-11-29 21:21:43,520 ERROR nodepool.NodeLauncher-0000318818: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318818: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:21:43,520 INFO nodepool.NodeLauncher-0000318818: Quota exceeded, invalidating quota cache 2018-11-29 21:21:43,521 INFO nodepool.NodeLauncher-0000318818: Aborting node 0000318818 due to quota failure 2018-11-29 21:21:44,868 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:45,162 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:21:45,538 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6696445941925049s 2018-11-29 21:21:46,326 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.1645739078521729s 2018-11-29 21:21:47,372 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:47,560 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318817 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:21:47,560 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:21:47,565 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318817, state=deleting, external_id=None 2018-11-29 21:21:48,452 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.0798773765563965s 2018-11-29 21:21:51,120 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:51,738 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6182210445404053s 2018-11-29 21:21:52,359 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:21:52,521 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:21:52,683 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.16147804260253906s 2018-11-29 21:21:52,683 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:21:52,684 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:21:52,823 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13897180557250977s 2018-11-29 21:21:52,826 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:53,089 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:21:53,091 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:21:53,092 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:21:53,093 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:21:53,094 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:21:53,096 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:21:53,102 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:21:53,103 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:21:53,103 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:21:53,104 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:21:53,105 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:21:53,105 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:21:53,106 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:21:53,106 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:21:53,198 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.37126612663269043s 2018-11-29 21:21:53,217 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:21:53,217 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:21:53,397 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:21:53,455 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:21:53,455 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:21:53,456 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:21:53,467 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318819 for request 300-0000331819 2018-11-29 21:21:53,476 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:21:53,477 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:21:53,478 INFO nodepool.NodeLauncher-0000318819: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318819 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318819 2018-11-29 21:21:53,478 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:21:53,721 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.356910228729248s 2018-11-29 21:21:53,737 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:54,519 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.7807004451751709s 2018-11-29 21:21:54,780 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.3016083240509033s 2018-11-29 21:21:54,780 ERROR nodepool.NodeLauncher-0000318819: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318819: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:21:54,781 INFO nodepool.NodeLauncher-0000318819: Quota exceeded, invalidating quota cache 2018-11-29 21:21:54,781 INFO nodepool.NodeLauncher-0000318819: Aborting node 0000318819 due to quota failure 2018-11-29 21:21:57,561 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:21:57,984 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.42352795600891113s 2018-11-29 21:21:58,336 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318818 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:21:58,336 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:21:58,340 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318818, state=deleting, external_id=None 2018-11-29 21:21:59,684 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:00,876 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.191192865371704s 2018-11-29 21:22:00,876 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:22:02,191 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.314485788345337s 2018-11-29 21:22:03,508 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:22:03,721 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.21273088455200195s 2018-11-29 21:22:03,722 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:22:03,722 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:22:03,759 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:03,828 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:22:03,834 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:22:03,834 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:22:03,835 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:22:03,835 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:22:03,838 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:22:03,839 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:22:03,839 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:22:03,840 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:22:03,840 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:22:03,841 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:22:03,842 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:22:03,843 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:22:03,843 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:22:03,862 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.13916254043579102s 2018-11-29 21:22:03,865 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:04,393 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5276803970336914s 2018-11-29 21:22:04,411 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:22:04,411 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:22:04,465 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.706352710723877s 2018-11-29 21:22:04,531 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:22:04,588 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:22:04,588 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:22:04,588 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:22:04,600 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318820 for request 300-0000331925 2018-11-29 21:22:04,606 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:22:04,607 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:22:04,608 INFO nodepool.NodeLauncher-0000318820: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318820 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318820 2018-11-29 21:22:04,609 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:22:05,829 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2202284336090088s 2018-11-29 21:22:05,829 ERROR nodepool.NodeLauncher-0000318820: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318820: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:22:05,830 INFO nodepool.NodeLauncher-0000318820: Quota exceeded, invalidating quota cache 2018-11-29 21:22:05,830 INFO nodepool.NodeLauncher-0000318820: Aborting node 0000318820 due to quota failure 2018-11-29 21:22:05,901 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:07,046 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1444342136383057s 2018-11-29 21:22:08,224 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:22:09,059 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318819 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:22:09,059 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:22:09,063 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318819, state=deleting, external_id=None 2018-11-29 21:22:09,525 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3009872436523438s 2018-11-29 21:22:10,009 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:10,631 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6207468509674072s 2018-11-29 21:22:11,523 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:11,879 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.35657310485839844s 2018-11-29 21:22:11,880 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-4c841ea4-b5b7-4c32-89a3-0cc3a4425611) 2018-11-29 21:22:11,996 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:22:12,187 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:13,704 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.5173866748809814s 2018-11-29 21:22:14,524 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:22:14,525 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:22:14,527 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:22:14,528 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:22:14,530 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:22:14,531 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:22:14,532 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:22:14,533 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:22:14,533 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:22:14,534 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:22:14,535 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:22:14,536 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:22:14,539 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:22:14,567 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:22:14,619 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:22:14,865 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.24589157104492188s 2018-11-29 21:22:14,865 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:22:14,866 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:22:15,073 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.2069694995880127s 2018-11-29 21:22:15,077 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:15,453 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.3764786720275879s 2018-11-29 21:22:15,471 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:22:15,471 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:22:15,553 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:22:15,595 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:22:15,649 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:22:15,650 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:22:15,650 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:22:15,659 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318821 for request 300-0000331819 2018-11-29 21:22:15,668 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:22:15,668 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:22:15,669 INFO nodepool.NodeLauncher-0000318821: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318821 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318821 2018-11-29 21:22:15,670 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:22:16,485 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:16,781 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2278552055358887s 2018-11-29 21:22:17,024 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.3545868396759033s 2018-11-29 21:22:17,025 ERROR nodepool.NodeLauncher-0000318821: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318821: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:22:17,025 INFO nodepool.NodeLauncher-0000318821: Quota exceeded, invalidating quota cache 2018-11-29 21:22:17,025 INFO nodepool.NodeLauncher-0000318821: Aborting node 0000318821 due to quota failure 2018-11-29 21:22:17,111 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6259641647338867s 2018-11-29 21:22:18,747 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:19,766 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318820 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:22:19,766 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:22:19,769 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318820, state=deleting, external_id=None 2018-11-29 21:22:19,911 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1638493537902832s 2018-11-29 21:22:22,654 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:22,812 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:22:23,261 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6062681674957275s 2018-11-29 21:22:23,984 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.172264814376831s 2018-11-29 21:22:24,930 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:25,251 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:22:25,253 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:22:25,255 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:22:25,256 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:22:25,257 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:22:25,257 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:22:25,258 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:22:25,258 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:22:25,258 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:22:25,259 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:22:25,259 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:22:25,259 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:22:25,260 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:22:25,260 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:22:25,672 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:22:25,791 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8604803085327148s 2018-11-29 21:22:25,905 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.23245692253112793s 2018-11-29 21:22:25,905 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:22:25,906 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:22:26,114 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.2076408863067627s 2018-11-29 21:22:26,117 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:26,766 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6486320495605469s 2018-11-29 21:22:26,783 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:22:26,784 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:22:26,885 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:22:26,932 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:22:26,933 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:22:26,933 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:22:26,943 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318822 for request 300-0000331925 2018-11-29 21:22:26,950 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:22:26,950 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:22:26,951 INFO nodepool.NodeLauncher-0000318822: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318822 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318822 2018-11-29 21:22:26,952 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:22:28,647 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.6950316429138184s 2018-11-29 21:22:28,647 ERROR nodepool.NodeLauncher-0000318822: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318822: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:22:28,648 INFO nodepool.NodeLauncher-0000318822: Quota exceeded, invalidating quota cache 2018-11-29 21:22:28,648 INFO nodepool.NodeLauncher-0000318822: Aborting node 0000318822 due to quota failure 2018-11-29 21:22:29,133 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:30,019 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:22:30,083 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.9499416351318359s 2018-11-29 21:22:30,471 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318821 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:22:30,471 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:22:30,476 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318821, state=deleting, external_id=None 2018-11-29 21:22:31,333 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.314237117767334s 2018-11-29 21:22:31,349 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:32,072 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.7236044406890869s 2018-11-29 21:22:35,293 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:36,047 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:22:36,049 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:22:36,050 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:22:36,052 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:22:36,057 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:22:36,057 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:22:36,058 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:22:36,058 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:22:36,058 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:22:36,059 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:22:36,060 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:22:36,060 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:22:36,061 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:22:36,061 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:22:36,174 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.8704907894134521s 2018-11-29 21:22:36,955 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:22:37,096 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.1407299041748047s 2018-11-29 21:22:37,097 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:22:37,097 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:22:37,361 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:37,400 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.3022644519805908s 2018-11-29 21:22:37,403 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:37,856 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.45334601402282715s 2018-11-29 21:22:37,874 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:22:37,877 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:22:37,990 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:22:38,038 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:22:38,038 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:22:38,038 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:22:38,049 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318823 for request 300-0000331819 2018-11-29 21:22:38,058 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:22:38,058 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:22:38,060 INFO nodepool.NodeLauncher-0000318823: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318823 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318823 2018-11-29 21:22:38,060 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:22:38,313 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9509086608886719s 2018-11-29 21:22:38,326 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:22:39,652 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3258581161499023s 2018-11-29 21:22:39,992 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.931901216506958s 2018-11-29 21:22:39,993 ERROR nodepool.NodeLauncher-0000318823: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318823: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:22:39,994 INFO nodepool.NodeLauncher-0000318823: Quota exceeded, invalidating quota cache 2018-11-29 21:22:39,995 INFO nodepool.NodeLauncher-0000318823: Aborting node 0000318823 due to quota failure 2018-11-29 21:22:41,481 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318822 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:22:41,481 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:22:41,485 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318822, state=deleting, external_id=None 2018-11-29 21:22:42,110 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:43,468 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:43,913 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.8031220436096191s 2018-11-29 21:22:44,937 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.468881607055664s 2018-11-29 21:22:45,682 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:22:46,827 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:22:46,827 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:22:46,829 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:22:46,829 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:22:46,830 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:22:46,831 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:22:46,831 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:22:46,831 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:22:46,833 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:22:46,833 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:22:46,834 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:22:46,834 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:22:46,835 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:22:46,836 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:22:46,952 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2702126502990723s 2018-11-29 21:22:48,063 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:22:48,210 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.14680123329162598s 2018-11-29 21:22:48,210 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:22:48,210 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:22:48,407 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.19629931449890137s 2018-11-29 21:22:48,410 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:48,804 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.39374279975891113s 2018-11-29 21:22:48,822 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:22:48,823 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:22:48,950 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:22:49,016 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:22:49,016 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:22:49,016 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:22:49,026 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318824 for request 300-0000331925 2018-11-29 21:22:49,034 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:22:49,034 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:22:49,035 INFO nodepool.NodeLauncher-0000318824: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318824 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318824 2018-11-29 21:22:49,036 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:22:49,929 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:49,975 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:50,476 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.4402625560760498s 2018-11-29 21:22:50,477 ERROR nodepool.NodeLauncher-0000318824: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318824: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:22:50,477 INFO nodepool.NodeLauncher-0000318824: Quota exceeded, invalidating quota cache 2018-11-29 21:22:50,477 INFO nodepool.NodeLauncher-0000318824: Aborting node 0000318824 due to quota failure 2018-11-29 21:22:51,102 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.17311429977417s 2018-11-29 21:22:52,058 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 2.0830626487731934s 2018-11-29 21:22:52,087 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318823 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:22:52,087 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:22:52,092 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318823, state=deleting, external_id=None 2018-11-29 21:22:52,983 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:22:54,302 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.3189377784729004s 2018-11-29 21:22:56,213 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:56,846 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6323103904724121s 2018-11-29 21:22:57,574 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:22:57,575 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:22:57,579 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:22:57,580 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:22:57,580 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:22:57,581 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:22:57,582 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:22:57,582 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:22:57,583 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:22:57,583 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:22:57,584 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:22:57,585 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:22:57,585 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:22:57,586 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:22:57,794 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:58,768 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.9738016128540039s 2018-11-29 21:22:59,039 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:22:59,295 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.2557709217071533s 2018-11-29 21:22:59,296 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:22:59,296 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:22:59,493 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.19713950157165527s 2018-11-29 21:22:59,497 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:22:59,830 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.3337697982788086s 2018-11-29 21:22:59,849 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:22:59,849 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:22:59,963 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:23:00,054 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:23:00,054 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:23:00,055 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:23:00,067 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318825 for request 300-0000331819 2018-11-29 21:23:00,075 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:23:00,075 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:23:00,079 INFO nodepool.NodeLauncher-0000318825: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318825 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318825 2018-11-29 21:23:00,080 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:23:00,330 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:23:01,279 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.1987769603729248s 2018-11-29 21:23:01,280 ERROR nodepool.NodeLauncher-0000318825: Request 300-0000331819: Launch attempt 1/3 failed for node 0000318825: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:23:01,280 INFO nodepool.NodeLauncher-0000318825: Quota exceeded, invalidating quota cache 2018-11-29 21:23:01,281 INFO nodepool.NodeLauncher-0000318825: Aborting node 0000318825 due to quota failure 2018-11-29 21:23:01,524 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.193739891052246s 2018-11-29 21:23:02,816 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318824 (state: aborted, allocated_to: 300-0000331925) 2018-11-29 21:23:02,816 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:23:02,819 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318824, state=deleting, external_id=None 2018-11-29 21:23:02,868 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:23:03,303 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.43457937240600586s 2018-11-29 21:23:03,862 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:23:04,713 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8502054214477539s 2018-11-29 21:23:07,557 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:23:08,310 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:23:08,312 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:23:08,313 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:23:08,318 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:23:08,319 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:23:08,319 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:23:08,319 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:23:08,320 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:23:08,321 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:23:08,323 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:23:08,323 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:23:08,324 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:23:08,324 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:23:08,327 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:23:09,122 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.5594208240509033s 2018-11-29 21:23:09,139 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:23:09,875 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:23:10,080 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:23:10,293 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.21309518814086914s 2018-11-29 21:23:10,294 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:23:10,294 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:23:10,308 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 1.168806552886963s 2018-11-29 21:23:10,421 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.1267099380493164s 2018-11-29 21:23:10,425 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:23:10,727 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 0.8517558574676514s 2018-11-29 21:23:11,089 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.6640903949737549s 2018-11-29 21:23:11,107 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:23:11,107 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331925 2018-11-29 21:23:11,220 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:23:11,268 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:23:11,269 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:23:11,269 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:23:11,279 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318826 for request 300-0000331925 2018-11-29 21:23:11,286 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331819 2018-11-29 21:23:11,286 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:23:11,288 INFO nodepool.NodeLauncher-0000318826: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318826 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318826 2018-11-29 21:23:11,289 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:23:12,384 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc running task ComputeGetServersDetail (queue 0) 2018-11-29 21:23:12,514 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputePostOs-volumes_boot in 1.2253026962280273s 2018-11-29 21:23:12,515 ERROR nodepool.NodeLauncher-0000318826: Request 300-0000331925: Launch attempt 1/3 failed for node 0000318826: Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 227, in launch self._launchNode() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/handler.py", line 140, in _launchNode instance_properties=self.label.instance_properties) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 329, in createServer return self._client.create_server(wait=False, **create_args) File "", line 2, in create_server File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/_utils.py", line 377, in func_wrapper return func(*args, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 6816, in create_server self.compute.post(endpoint, json=server_json)) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 95, in _json_response exceptions.raise_from_response(response, error_message=error_message) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://compute-ca-ymq-1.vexxhost.net/v2.1/os-volumes_boot, {"forbidden": {"message": "Quota exceeded for cores: Requested 1, but already used 8 of 8 cores", "code": 403}} 2018-11-29 21:23:12,515 INFO nodepool.NodeLauncher-0000318826: Quota exceeded, invalidating quota cache 2018-11-29 21:23:12,516 INFO nodepool.NodeLauncher-0000318826: Aborting node 0000318826 due to quota failure 2018-11-29 21:23:12,784 DEBUG nodepool.TaskManager: Manager limestone-ansible-network-us-slc ran task ComputeGetServersDetail in 0.4002060890197754s 2018-11-29 21:23:12,785 ERROR nodepool.CleanupWorker: Exception in CleanupWorker (leaked instance cleanup) Traceback (most recent call last): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 598, in _run task() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/launcher.py", line 474, in _cleanupLeakedInstances manager.cleanupLeakedResources() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 465, in cleanupLeakedResources for server in self.listNodes(): File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/driver/openstack/provider.py", line 434, in listNodes return self._client.list_servers() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2111, in list_servers filters=filters) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/cloud/openstackcloud.py", line 2128, in _list_servers '/servers/detail', params=params), File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 320, in get return self.request(url, 'GET', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/_adapter.py", line 145, in request **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 149, in submit_function return self.submit_task(task) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/nodepool/task_manager.py", line 105, in submit_task return task.wait() File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 79, in wait self._traceback) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/six.py", line 686, in reraise raise value File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 87, in run self.done(self.main()) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/openstack/task_manager.py", line 59, in main return self._main(*self.args, **self.kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 205, in request return self.session.request(url, method, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 684, in request auth_headers = self.get_auth_headers(auth) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers return auth.get_headers(self, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/plugin.py", line 95, in get_headers token = self.get_token(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref authenticated=False, log=False, **rkwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 1019, in post return self.request(url, 'POST', **kwargs) File "/opt/rh/rh-python35/root/usr/lib/python3.5/site-packages/keystoneauth1/session.py", line 869, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-54de21d8-ea87-40d0-b573-0f828ceed080) 2018-11-29 21:23:12,913 DEBUG nodepool.CleanupWorker: Cleaning up held nodes... 2018-11-29 21:23:13,529 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000318825 (state: aborted, allocated_to: 300-0000331819) 2018-11-29 21:23:13,529 INFO nodepool.DeletedNodeWorker: Deleting aborted instance None from vexxhost-ansible-network-mtl1 2018-11-29 21:23:13,532 INFO nodepool.NodeDeleter: Deleting ZK node id=0000318825, state=deleting, external_id=None 2018-11-29 21:23:15,150 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:23:15,329 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:23:15,998 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.6686029434204102s 2018-11-29 21:23:16,367 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.2164306640625s 2018-11-29 21:23:16,382 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0) 2018-11-29 21:23:17,545 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task ComputeGetServersDetail in 1.1602551937103271s 2018-11-29 21:23:19,047 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.small: Active requests: [] 2018-11-29 21:23:19,049 DEBUG nodepool.PoolWorker.managed-runc-provider-oci01.softwarefactory-project.io: Active requests: [] 2018-11-29 21:23:19,049 DEBUG nodepool.PoolWorker.vexxhost-rdo-ca-ymq-1-main: Active requests: [] 2018-11-29 21:23:19,050 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org-tripleo: Active requests: [] 2018-11-29 21:23:19,050 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-slc-s1.large: Active requests: [] 2018-11-29 21:23:19,051 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.small: Active requests: [] 2018-11-29 21:23:19,051 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-4: Active requests: [] 2018-11-29 21:23:19,052 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-sjc1-v2-highcpu-1: Active requests: [] 2018-11-29 21:23:19,052 DEBUG nodepool.PoolWorker.rdo-cloud-rdoproject.org: Active requests: [] 2018-11-29 21:23:19,053 DEBUG nodepool.PoolWorker.rdo-cloud-tripleo-ovb: Active requests: [] 2018-11-29 21:23:19,053 DEBUG nodepool.PoolWorker.limestone-ansible-network-us-dfw-1-s1.large: Active requests: [] 2018-11-29 21:23:19,055 DEBUG nodepool.PoolWorker.rdo-cloud-main: Active requests: [] 2018-11-29 21:23:19,056 DEBUG nodepool.PoolWorker.ansible-statics-main: Active requests: [] 2018-11-29 21:23:19,056 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-4: Active requests: [] 2018-11-29 21:23:21,292 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetLimits (queue 0) 2018-11-29 21:23:21,439 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetLimits in 0.14704585075378418s 2018-11-29 21:23:21,440 DEBUG nodepool.driver.openstack.OpenStackProvider: Provider quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:23:21,440 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetFlavorsDetail (queue 0) 2018-11-29 21:23:21,659 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetFlavorsDetail in 0.21837425231933594s 2018-11-29 21:23:21,663 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputeGetServersDetail (queue 0) 2018-11-29 21:23:22,011 DEBUG nodepool.TaskManager: Manager rdo-cloud running task ComputeGetServersDetail (queue 0) 2018-11-29 21:23:22,225 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 ran task ComputeGetServersDetail in 0.5623042583465576s 2018-11-29 21:23:22,243 DEBUG nodepool.driver.openstack.OpenStackProvider: Available quota for vexxhost-ansible-network-mtl1: {'compute': {'cores': 8, 'ram': 51200, 'instances': 10}} 2018-11-29 21:23:22,243 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Retrying node request 300-0000331819 2018-11-29 21:23:22,365 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining provider quota: {'compute': {'cores': 6, 'ram': 49152, 'instances': 8}} 2018-11-29 21:23:22,399 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task NetworkGetPorts (queue 0) 2018-11-29 21:23:22,422 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Current pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 7}} 2018-11-29 21:23:22,422 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Predicted remaining pool quota: {'compute': {'cores': inf, 'ram': inf, 'instances': 6}} 2018-11-29 21:23:22,422 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Unpaused request 2018-11-29 21:23:22,435 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Locked building node 0000318827 for request 300-0000331819 2018-11-29 21:23:22,442 DEBUG nodepool.driver.NodeRequestHandler[zs.softwarefactory-project.io-2296-PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1]: Pausing request handling after node abort to satisfy request 300-0000331925 2018-11-29 21:23:22,442 DEBUG nodepool.PoolWorker.vexxhost-ansible-network-mtl1-v2-highcpu-1: Active requests: ['300-0000331819', '300-0000331925'] 2018-11-29 21:23:22,444 INFO nodepool.NodeLauncher-0000318827: Creating server with hostname ansible-fedora-28-1vcpu-vexxhost-mtl1-0000318827 in vexxhost-ansible-network-mtl1 from image ansible-fedora-28 for node id: 0000318827 2018-11-29 21:23:22,445 DEBUG nodepool.TaskManager: Manager vexxhost-ansible-network-mtl1 running task ComputePostOs-volumes_boot (queue 0) 2018-11-29 21:23:22,778 DEBUG nodepool.TaskManager: Manager rdo-cloud ran task ComputeGetServersDetail in 0.7661676406860352s 2018-11-29 21:23:23,970 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo ran task NetworkGetPorts in 1.5715129375457764s 2018-11-29 21:23:23,987 DEBUG nodepool.TaskManager: Manager rdo-cloud-tripleo running task ComputeGetServersDetail (queue 0)