2026-03-17 15:08:00,856 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.66.245:42995

2026-03-17 15:08:00,856 - distributed.scheduler - INFO - Register worker addr: tcp://192.168.66.245:42995 name: simple-default-worker-b093fed4b1

2026-03-17 15:07:48,220 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.184.244:44713

2026-03-17 15:07:48,219 - distributed.scheduler - INFO - Register worker addr: tcp://192.168.184.244:44713 name: simple-default-worker-27f2320dff

2026-03-17 14:53:03,754 - distributed.scheduler - ERROR - 1/1 nanny worker(s) did not shut down within 30s: {'tcp://192.168.184.244:34957'} Traceback (most recent call last): File "/opt/conda/lib/python3.12/site-packages/distributed/utils.py", line 818, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.12/site-packages/distributed/scheduler.py", line 8729, in check_worker_ttl await self.restart_workers( File "/opt/conda/lib/python3.12/site-packages/distributed/utils.py", line 818, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.12/site-packages/distributed/scheduler.py", line 6666, in restart_workers raise TimeoutError( TimeoutError: 1/1 nanny worker(s) did not shut down within 30s: {'tcp://192.168.184.244:34957'}

2026-03-17 14:53:03,753 - distributed.scheduler - ERROR - 1/1 nanny worker(s) did not shut down within 30s: {'tcp://192.168.184.244:34957'} Traceback (most recent call last): File "/opt/conda/lib/python3.12/site-packages/distributed/utils.py", line 818, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.12/site-packages/distributed/scheduler.py", line 6666, in restart_workers raise TimeoutError( TimeoutError: 1/1 nanny worker(s) did not shut down within 30s: {'tcp://192.168.184.244:34957'}

2026-03-17 14:53:03,752 - distributed.scheduler - INFO - Remove worker addr: tcp://192.168.184.244:34957 name: simple-default-worker-27f2320dff (stimulus_id='check-worker-ttl-1773759153.7511811')

2026-03-17 14:53:03,752 - distributed.scheduler - ERROR - Workers ['tcp://192.168.184.244:34957'] did not shut down within 30s; force closing

2026-03-17 14:52:33,751 - distributed.scheduler - INFO - Restarting 1 workers: ['tcp://192.168.184.244:34957'] (stimulus_id='check-worker-ttl-1773759153.7511811'

2026-03-17 14:52:33,751 - distributed.scheduler - WARNING - Worker failed to heartbeat for 333s; attempting restart: <WorkerState 'tcp://192.168.184.244:34957', name: simple-default-worker-27f2320dff, status: running, memory: 0, processing: 0>

2026-03-17 14:39:40,725 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.184.244:34957

2026-03-17 14:39:40,725 - distributed.scheduler - INFO - Register worker addr: tcp://192.168.184.244:34957 name: simple-default-worker-27f2320dff

2026-03-17 14:38:17,125 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://192.168.184.244:38665'.

2026-03-17 14:38:03,755 - distributed.scheduler - ERROR - 1/1 nanny worker(s) did not shut down within 30s: {'tcp://192.168.184.244:38665'} Traceback (most recent call last): File "/opt/conda/lib/python3.12/site-packages/distributed/utils.py", line 818, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.12/site-packages/distributed/scheduler.py", line 8729, in check_worker_ttl await self.restart_workers( File "/opt/conda/lib/python3.12/site-packages/distributed/utils.py", line 818, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.12/site-packages/distributed/scheduler.py", line 6666, in restart_workers raise TimeoutError( TimeoutError: 1/1 nanny worker(s) did not shut down within 30s: {'tcp://192.168.184.244:38665'}

2026-03-17 14:38:03,755 - distributed.scheduler - ERROR - 1/1 nanny worker(s) did not shut down within 30s: {'tcp://192.168.184.244:38665'} Traceback (most recent call last): File "/opt/conda/lib/python3.12/site-packages/distributed/utils.py", line 818, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.12/site-packages/distributed/scheduler.py", line 6666, in restart_workers raise TimeoutError( TimeoutError: 1/1 nanny worker(s) did not shut down within 30s: {'tcp://192.168.184.244:38665'}

2026-03-17 14:38:03,753 - distributed.scheduler - INFO - Remove worker addr: tcp://192.168.184.244:38665 name: simple-default-worker-27f2320dff (stimulus_id='check-worker-ttl-1773758253.7513306')

2026-03-17 14:38:03,753 - distributed.scheduler - ERROR - Workers ['tcp://192.168.184.244:38665'] did not shut down within 30s; force closing

2026-03-17 14:37:33,751 - distributed.scheduler - INFO - Restarting 1 workers: ['tcp://192.168.184.244:38665'] (stimulus_id='check-worker-ttl-1773758253.7513306'

2026-03-17 14:37:33,751 - distributed.scheduler - WARNING - Worker failed to heartbeat for 394s; attempting restart: <WorkerState 'tcp://192.168.184.244:38665', name: simple-default-worker-27f2320dff, status: running, memory: 0, processing: 0>

2026-03-17 14:13:03,762 - distributed.scheduler - ERROR - 1/1 nanny worker(s) did not shut down within 30s: {'tcp://192.168.66.245:36465'} Traceback (most recent call last): File "/opt/conda/lib/python3.12/site-packages/distributed/utils.py", line 818, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.12/site-packages/distributed/scheduler.py", line 8729, in check_worker_ttl await self.restart_workers( File "/opt/conda/lib/python3.12/site-packages/distributed/utils.py", line 818, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.12/site-packages/distributed/scheduler.py", line 6666, in restart_workers raise TimeoutError( TimeoutError: 1/1 nanny worker(s) did not shut down within 30s: {'tcp://192.168.66.245:36465'}

2026-03-17 14:13:03,761 - distributed.scheduler - ERROR - 1/1 nanny worker(s) did not shut down within 30s: {'tcp://192.168.66.245:36465'} Traceback (most recent call last): File "/opt/conda/lib/python3.12/site-packages/distributed/utils.py", line 818, in wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.12/site-packages/distributed/scheduler.py", line 6666, in restart_workers raise TimeoutError( TimeoutError: 1/1 nanny worker(s) did not shut down within 30s: {'tcp://192.168.66.245:36465'}

2026-03-17 14:13:03,754 - distributed.scheduler - INFO - Remove worker addr: tcp://192.168.66.245:36465 name: simple-default-worker-b093fed4b1 (stimulus_id='check-worker-ttl-1773756753.7520256')

2026-03-17 14:13:03,753 - distributed.scheduler - ERROR - Workers ['tcp://192.168.66.245:36465'] did not shut down within 30s; force closing

2026-03-17 14:12:33,753 - distributed.scheduler - INFO - Restarting 1 workers: ['tcp://192.168.66.245:36465'] (stimulus_id='check-worker-ttl-1773756753.7520256'

2026-03-17 14:12:33,752 - distributed.scheduler - WARNING - Worker failed to heartbeat for 372s; attempting restart: <WorkerState 'tcp://192.168.66.245:36465', name: simple-default-worker-b093fed4b1, status: running, memory: 0, processing: 0>

2026-02-17 17:16:59,439 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.66.245:36465

2026-02-17 17:16:59,439 - distributed.scheduler - INFO - Register worker addr: tcp://192.168.66.245:36465 name: simple-default-worker-b093fed4b1

2026-02-17 17:16:55,089 - distributed.scheduler - INFO - Retired worker 'tcp://192.168.242.175:36871' (stimulus_id='retire-workers-1771348615.0883086')

2026-02-17 17:16:55,089 - distributed.scheduler - INFO - Remove worker addr: tcp://192.168.242.175:36871 name: simple-default-worker-bcd45c38a1 (stimulus_id='retire-workers-1771348615.0883086')

2026-02-17 17:16:55,089 - distributed.scheduler - INFO - Retired worker 'tcp://192.168.66.219:40819' (stimulus_id='retire-workers-1771348615.0883086')

2026-02-17 17:16:55,088 - distributed.scheduler - INFO - Remove worker addr: tcp://192.168.66.219:40819 name: simple-default-worker-27ac8ccc9d (stimulus_id='retire-workers-1771348615.0883086')

2026-02-17 17:16:55,088 - distributed.scheduler - INFO - Retire worker names ('simple-default-worker-27ac8ccc9d', 'simple-default-worker-bcd45c38a1')

2026-02-17 12:48:31,827 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.184.244:38665

2026-02-17 12:48:31,827 - distributed.scheduler - INFO - Register worker addr: tcp://192.168.184.244:38665 name: simple-default-worker-27f2320dff

2026-02-17 12:47:39,759 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.66.219:40819

2026-02-17 12:47:39,758 - distributed.scheduler - INFO - Register worker addr: tcp://192.168.66.219:40819 name: simple-default-worker-27ac8ccc9d

2026-02-17 12:47:39,643 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.242.175:36871

2026-02-17 12:47:39,643 - distributed.scheduler - INFO - Register worker addr: tcp://192.168.242.175:36871 name: simple-default-worker-bcd45c38a1

2026-02-17 12:47:38,755 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.242.129:41501

2026-02-17 12:47:38,749 - distributed.scheduler - INFO - Register worker addr: tcp://192.168.242.129:41501 name: simple-default-worker-97d05d9862

2026-02-17 12:47:33,749 - distributed.scheduler - INFO - Registering Worker plugin shuffle

2026-02-17 12:47:33,749 - distributed.scheduler - INFO - dashboard at: http://192.168.242.149:8787/status

2026-02-17 12:47:33,749 - distributed.scheduler - INFO - Scheduler at: tcp://192.168.242.149:8786

2026-02-17 12:47:33,748 - distributed.scheduler - INFO - -----------------------------------------------

2026-02-17 12:47:33,567 - distributed.scheduler - INFO - State start