-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathlog_k10.txt
5129 lines (5129 loc) · 511 KB
/
log_k10.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
2024-04-12 20:05:06 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-04-12 20:05:06 INFO kmeans$:36 - Input: project-q1-vectors/part-00000
2024-04-12 20:05:06 INFO kmeans$:37 - Output path: project-q1-kmeans
2024-04-12 20:05:06 INFO kmeans$:38 - Number of clusters: 10
2024-04-12 20:05:06 INFO SparkContext:54 - Running Spark version 2.3.1
2024-04-12 20:05:07 INFO SparkContext:54 - Submitted application: KMeansproject-q1-vectors/part-00000
2024-04-12 20:05:07 INFO SecurityManager:54 - Changing view acls to: rithickumar
2024-04-12 20:05:07 INFO SecurityManager:54 - Changing modify acls to: rithickumar
2024-04-12 20:05:07 INFO SecurityManager:54 - Changing view acls groups to:
2024-04-12 20:05:07 INFO SecurityManager:54 - Changing modify acls groups to:
2024-04-12 20:05:07 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rithickumar); groups with view permissions: Set(); users with modify permissions: Set(rithickumar); groups with modify permissions: Set()
2024-04-12 20:05:12 INFO Utils:54 - Successfully started service 'sparkDriver' on port 53057.
2024-04-12 20:05:12 INFO SparkEnv:54 - Registering MapOutputTracker
2024-04-12 20:05:12 INFO SparkEnv:54 - Registering BlockManagerMaster
2024-04-12 20:05:12 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2024-04-12 20:05:12 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2024-04-12 20:05:12 INFO DiskBlockManager:54 - Created local directory at /private/var/folders/jp/l319lw651tb48gz1j9rnkrjm0000gn/T/blockmgr-4898a257-d59b-4a3b-93d6-300420a1840a
2024-04-12 20:05:12 INFO MemoryStore:54 - MemoryStore started with capacity 366.3 MB
2024-04-12 20:05:12 INFO SparkEnv:54 - Registering OutputCommitCoordinator
2024-04-12 20:05:12 INFO log:192 - Logging initialized @17351ms
2024-04-12 20:05:12 INFO Server:346 - jetty-9.3.z-SNAPSHOT
2024-04-12 20:05:12 INFO Server:414 - Started @17419ms
2024-04-12 20:05:12 INFO AbstractConnector:278 - Started ServerConnector@2179635f{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2024-04-12 20:05:12 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4837595f{/jobs,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5dda14d0{/jobs/json,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1db0ec27{/jobs/job,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@d4ab71a{/jobs/job/json,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3b4ef7{/stages,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1af05b03{/stages/json,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5987e932{/stages/stage,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@438bad7c{/stages/stage/json,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@25230246{/stages/pool,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4fdf8f12{/stages/pool/json,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4a8b5227{/storage,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54f5f647{/storage/json,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6979efad{/storage/rdd,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5a6d5a8f{/storage/rdd/json,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4a67318f{/environment,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@315ba14a{/environment/json,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@17f9344b{/executors,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@27f0ad19{/executors/json,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54e81b21{/executors/threadDump,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@38d5b107{/executors/threadDump/json,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6650813a{/static,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77e2a6e2{/,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5710768a{/api,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@30272916{/jobs/job/kill,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5bb3d42d{/stages/stage/kill,null,AVAILABLE,@Spark}
2024-04-12 20:05:12 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://10.0.0.154:4040
2024-04-12 20:05:12 INFO SparkContext:54 - Added JAR file:/Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/target/assignments-1.0.jar at spark://10.0.0.154:53057/jars/assignments-1.0.jar with timestamp 1712966712790
2024-04-12 20:05:12 INFO Executor:54 - Starting executor ID driver on host localhost
2024-04-12 20:05:12 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 53058.
2024-04-12 20:05:12 INFO NettyBlockTransferService:54 - Server created on 10.0.0.154:53058
2024-04-12 20:05:12 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2024-04-12 20:05:12 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, 10.0.0.154, 53058, None)
2024-04-12 20:05:12 INFO BlockManagerMasterEndpoint:54 - Registering block manager 10.0.0.154:53058 with 366.3 MB RAM, BlockManagerId(driver, 10.0.0.154, 53058, None)
2024-04-12 20:05:12 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, 10.0.0.154, 53058, None)
2024-04-12 20:05:12 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, 10.0.0.154, 53058, None)
2024-04-12 20:05:13 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5eb97ced{/metrics/json,null,AVAILABLE,@Spark}
2024-04-12 20:05:13 WARN SparkContext:66 - Using an existing SparkContext; some configuration may not take effect.
2024-04-12 20:05:13 INFO SharedState:54 - Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/spark-warehouse/').
2024-04-12 20:05:13 INFO SharedState:54 - Warehouse path is 'file:/Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/spark-warehouse/'.
2024-04-12 20:05:13 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@280d9edc{/SQL,null,AVAILABLE,@Spark}
2024-04-12 20:05:13 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@28fd3dc1{/SQL/json,null,AVAILABLE,@Spark}
2024-04-12 20:05:13 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@15e0fe05{/SQL/execution,null,AVAILABLE,@Spark}
2024-04-12 20:05:13 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1128620c{/SQL/execution/json,null,AVAILABLE,@Spark}
2024-04-12 20:05:13 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@bdd2027{/static/sql,null,AVAILABLE,@Spark}
2024-04-12 20:05:13 INFO StateStoreCoordinatorRef:54 - Registered StateStoreCoordinator endpoint
2024-04-12 20:05:16 INFO FileSourceStrategy:54 - Pruning directories with:
2024-04-12 20:05:16 INFO FileSourceStrategy:54 - Post-Scan Filters: (length(trim(value#0, None)) > 0)
2024-04-12 20:05:16 INFO FileSourceStrategy:54 - Output Data Schema: struct<value: string>
2024-04-12 20:05:16 INFO FileSourceScanExec:54 - Pushed Filters:
2024-04-12 20:05:16 INFO CodeGenerator:54 - Code generated in 289.762625 ms
2024-04-12 20:05:17 INFO CodeGenerator:54 - Code generated in 68.53 ms
2024-04-12 20:05:17 INFO MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 277.7 KB, free 366.0 MB)
2024-04-12 20:05:17 INFO MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.5 KB, free 366.0 MB)
2024-04-12 20:05:17 INFO BlockManagerInfo:54 - Added broadcast_0_piece0 in memory on 10.0.0.154:53058 (size: 23.5 KB, free: 366.3 MB)
2024-04-12 20:05:17 INFO SparkContext:54 - Created broadcast 0 from csv at kmeans.scala:55
2024-04-12 20:05:17 INFO FileSourceScanExec:54 - Planning scan with bin packing, max size: 57073876 bytes, open cost is considered as scanning 4194304 bytes.
2024-04-12 20:05:17 INFO SparkContext:54 - Starting job: csv at kmeans.scala:55
2024-04-12 20:05:17 INFO DAGScheduler:54 - Got job 0 (csv at kmeans.scala:55) with 1 output partitions
2024-04-12 20:05:17 INFO DAGScheduler:54 - Final stage: ResultStage 0 (csv at kmeans.scala:55)
2024-04-12 20:05:17 INFO DAGScheduler:54 - Parents of final stage: List()
2024-04-12 20:05:17 INFO DAGScheduler:54 - Missing parents: List()
2024-04-12 20:05:17 INFO DAGScheduler:54 - Submitting ResultStage 0 (MapPartitionsRDD[3] at csv at kmeans.scala:55), which has no missing parents
2024-04-12 20:05:17 INFO MemoryStore:54 - Block broadcast_1 stored as values in memory (estimated size 9.4 KB, free 366.0 MB)
2024-04-12 20:05:17 INFO MemoryStore:54 - Block broadcast_1_piece0 stored as bytes in memory (estimated size 4.5 KB, free 366.0 MB)
2024-04-12 20:05:17 INFO BlockManagerInfo:54 - Added broadcast_1_piece0 in memory on 10.0.0.154:53058 (size: 4.5 KB, free: 366.3 MB)
2024-04-12 20:05:17 INFO SparkContext:54 - Created broadcast 1 from broadcast at DAGScheduler.scala:1039
2024-04-12 20:05:18 INFO DAGScheduler:54 - Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at csv at kmeans.scala:55) (first 15 tasks are for partitions Vector(0))
2024-04-12 20:05:18 INFO TaskSchedulerImpl:54 - Adding task set 0.0 with 1 tasks
2024-04-12 20:05:18 INFO TaskSetManager:54 - Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 8372 bytes)
2024-04-12 20:05:18 INFO Executor:54 - Running task 0.0 in stage 0.0 (TID 0)
2024-04-12 20:05:18 INFO Executor:54 - Fetching spark://10.0.0.154:53057/jars/assignments-1.0.jar with timestamp 1712966712790
2024-04-12 20:05:18 INFO TransportClientFactory:267 - Successfully created connection to /10.0.0.154:53057 after 35 ms (0 ms spent in bootstraps)
2024-04-12 20:05:18 INFO Utils:54 - Fetching spark://10.0.0.154:53057/jars/assignments-1.0.jar to /private/var/folders/jp/l319lw651tb48gz1j9rnkrjm0000gn/T/spark-7abc7798-023c-4602-b3eb-e0c2f3e417dd/userFiles-2975bf9e-5126-445f-8dd1-b5a0ccd96918/fetchFileTemp7161147041944033212.tmp
2024-04-12 20:05:18 INFO Executor:54 - Adding file:/private/var/folders/jp/l319lw651tb48gz1j9rnkrjm0000gn/T/spark-7abc7798-023c-4602-b3eb-e0c2f3e417dd/userFiles-2975bf9e-5126-445f-8dd1-b5a0ccd96918/assignments-1.0.jar to class loader
2024-04-12 20:05:18 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 0-57073876, partition values: [empty row]
2024-04-12 20:05:18 INFO CodeGenerator:54 - Code generated in 24.234708 ms
2024-04-12 20:05:19 INFO Executor:54 - Finished task 0.0 in stage 0.0 (TID 0). 1509 bytes result sent to driver
2024-04-12 20:05:19 INFO TaskSetManager:54 - Finished task 0.0 in stage 0.0 (TID 0) in 1329 ms on localhost (executor driver) (1/1)
2024-04-12 20:05:19 INFO TaskSchedulerImpl:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool
2024-04-12 20:05:19 INFO DAGScheduler:54 - ResultStage 0 (csv at kmeans.scala:55) finished in 1.474 s
2024-04-12 20:05:19 INFO DAGScheduler:54 - Job 0 finished: csv at kmeans.scala:55, took 1.520131 s
2024-04-12 20:05:19 INFO FileSourceStrategy:54 - Pruning directories with:
2024-04-12 20:05:19 INFO FileSourceStrategy:54 - Post-Scan Filters:
2024-04-12 20:05:19 INFO FileSourceStrategy:54 - Output Data Schema: struct<value: string>
2024-04-12 20:05:19 INFO FileSourceScanExec:54 - Pushed Filters:
2024-04-12 20:05:19 INFO CodeGenerator:54 - Code generated in 16.800542 ms
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 14
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 20
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 22
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 8
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 23
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 26
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 15
2024-04-12 20:05:19 INFO BlockManagerInfo:54 - Removed broadcast_1_piece0 on 10.0.0.154:53058 in memory (size: 4.5 KB, free: 366.3 MB)
2024-04-12 20:05:19 INFO MemoryStore:54 - Block broadcast_2 stored as values in memory (estimated size 277.7 KB, free 365.7 MB)
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 12
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 17
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 7
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 10
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 19
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 30
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 13
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 27
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 28
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 29
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 11
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 16
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 6
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 21
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 9
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 24
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 25
2024-04-12 20:05:19 INFO ContextCleaner:54 - Cleaned accumulator 18
2024-04-12 20:05:19 INFO MemoryStore:54 - Block broadcast_2_piece0 stored as bytes in memory (estimated size 23.5 KB, free 365.7 MB)
2024-04-12 20:05:19 INFO BlockManagerInfo:54 - Added broadcast_2_piece0 in memory on 10.0.0.154:53058 (size: 23.5 KB, free: 366.3 MB)
2024-04-12 20:05:19 INFO SparkContext:54 - Created broadcast 2 from csv at kmeans.scala:55
2024-04-12 20:05:19 INFO FileSourceScanExec:54 - Planning scan with bin packing, max size: 57073876 bytes, open cost is considered as scanning 4194304 bytes.
2024-04-12 20:05:19 INFO SparkContext:54 - Starting job: csv at kmeans.scala:55
2024-04-12 20:05:19 INFO DAGScheduler:54 - Got job 1 (csv at kmeans.scala:55) with 8 output partitions
2024-04-12 20:05:19 INFO DAGScheduler:54 - Final stage: ResultStage 1 (csv at kmeans.scala:55)
2024-04-12 20:05:19 INFO DAGScheduler:54 - Parents of final stage: List()
2024-04-12 20:05:19 INFO DAGScheduler:54 - Missing parents: List()
2024-04-12 20:05:19 INFO DAGScheduler:54 - Submitting ResultStage 1 (MapPartitionsRDD[8] at csv at kmeans.scala:55), which has no missing parents
2024-04-12 20:05:19 INFO MemoryStore:54 - Block broadcast_3 stored as values in memory (estimated size 14.8 KB, free 365.7 MB)
2024-04-12 20:05:19 INFO MemoryStore:54 - Block broadcast_3_piece0 stored as bytes in memory (estimated size 8.7 KB, free 365.7 MB)
2024-04-12 20:05:19 INFO BlockManagerInfo:54 - Added broadcast_3_piece0 in memory on 10.0.0.154:53058 (size: 8.7 KB, free: 366.2 MB)
2024-04-12 20:05:19 INFO SparkContext:54 - Created broadcast 3 from broadcast at DAGScheduler.scala:1039
2024-04-12 20:05:19 INFO DAGScheduler:54 - Submitting 8 missing tasks from ResultStage 1 (MapPartitionsRDD[8] at csv at kmeans.scala:55) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7))
2024-04-12 20:05:19 INFO TaskSchedulerImpl:54 - Adding task set 1.0 with 8 tasks
2024-04-12 20:05:19 INFO TaskSetManager:54 - Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0, PROCESS_LOCAL, 8372 bytes)
2024-04-12 20:05:19 INFO TaskSetManager:54 - Starting task 1.0 in stage 1.0 (TID 2, localhost, executor driver, partition 1, PROCESS_LOCAL, 8372 bytes)
2024-04-12 20:05:19 INFO TaskSetManager:54 - Starting task 2.0 in stage 1.0 (TID 3, localhost, executor driver, partition 2, PROCESS_LOCAL, 8372 bytes)
2024-04-12 20:05:19 INFO TaskSetManager:54 - Starting task 3.0 in stage 1.0 (TID 4, localhost, executor driver, partition 3, PROCESS_LOCAL, 8372 bytes)
2024-04-12 20:05:19 INFO TaskSetManager:54 - Starting task 4.0 in stage 1.0 (TID 5, localhost, executor driver, partition 4, PROCESS_LOCAL, 8372 bytes)
2024-04-12 20:05:19 INFO TaskSetManager:54 - Starting task 5.0 in stage 1.0 (TID 6, localhost, executor driver, partition 5, PROCESS_LOCAL, 8372 bytes)
2024-04-12 20:05:19 INFO TaskSetManager:54 - Starting task 6.0 in stage 1.0 (TID 7, localhost, executor driver, partition 6, PROCESS_LOCAL, 8372 bytes)
2024-04-12 20:05:19 INFO TaskSetManager:54 - Starting task 7.0 in stage 1.0 (TID 8, localhost, executor driver, partition 7, PROCESS_LOCAL, 8372 bytes)
2024-04-12 20:05:19 INFO Executor:54 - Running task 0.0 in stage 1.0 (TID 1)
2024-04-12 20:05:19 INFO Executor:54 - Running task 2.0 in stage 1.0 (TID 3)
2024-04-12 20:05:19 INFO Executor:54 - Running task 1.0 in stage 1.0 (TID 2)
2024-04-12 20:05:19 INFO Executor:54 - Running task 3.0 in stage 1.0 (TID 4)
2024-04-12 20:05:19 INFO Executor:54 - Running task 4.0 in stage 1.0 (TID 5)
2024-04-12 20:05:19 INFO Executor:54 - Running task 5.0 in stage 1.0 (TID 6)
2024-04-12 20:05:19 INFO Executor:54 - Running task 6.0 in stage 1.0 (TID 7)
2024-04-12 20:05:19 INFO Executor:54 - Running task 7.0 in stage 1.0 (TID 8)
2024-04-12 20:05:19 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 342443256-399517132, partition values: [empty row]
2024-04-12 20:05:19 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 399517132-452396709, partition values: [empty row]
2024-04-12 20:05:19 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 285369380-342443256, partition values: [empty row]
2024-04-12 20:05:19 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 171221628-228295504, partition values: [empty row]
2024-04-12 20:05:19 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 114147752-171221628, partition values: [empty row]
2024-04-12 20:05:19 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 228295504-285369380, partition values: [empty row]
2024-04-12 20:05:19 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 0-57073876, partition values: [empty row]
2024-04-12 20:05:19 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 57073876-114147752, partition values: [empty row]
2024-04-12 20:05:20 INFO ContextCleaner:54 - Cleaned accumulator 0
2024-04-12 20:05:20 INFO ContextCleaner:54 - Cleaned accumulator 3
2024-04-12 20:05:20 INFO ContextCleaner:54 - Cleaned accumulator 1
2024-04-12 20:05:20 INFO ContextCleaner:54 - Cleaned accumulator 5
2024-04-12 20:05:20 INFO ContextCleaner:54 - Cleaned accumulator 4
2024-04-12 20:05:20 INFO ContextCleaner:54 - Cleaned accumulator 2
2024-04-12 20:05:21 INFO BlockManagerInfo:54 - Removed broadcast_0_piece0 on 10.0.0.154:53058 in memory (size: 23.5 KB, free: 366.3 MB)
2024-04-12 20:05:23 INFO Executor:54 - Finished task 7.0 in stage 1.0 (TID 8). 1599 bytes result sent to driver
2024-04-12 20:05:23 INFO TaskSetManager:54 - Finished task 7.0 in stage 1.0 (TID 8) in 4234 ms on localhost (executor driver) (1/8)
2024-04-12 20:05:24 INFO Executor:54 - Finished task 6.0 in stage 1.0 (TID 7). 1599 bytes result sent to driver
2024-04-12 20:05:24 INFO Executor:54 - Finished task 1.0 in stage 1.0 (TID 2). 1599 bytes result sent to driver
2024-04-12 20:05:24 INFO Executor:54 - Finished task 3.0 in stage 1.0 (TID 4). 1599 bytes result sent to driver
2024-04-12 20:05:24 INFO Executor:54 - Finished task 4.0 in stage 1.0 (TID 5). 1599 bytes result sent to driver
2024-04-12 20:05:24 INFO TaskSetManager:54 - Finished task 6.0 in stage 1.0 (TID 7) in 4425 ms on localhost (executor driver) (2/8)
2024-04-12 20:05:24 INFO TaskSetManager:54 - Finished task 3.0 in stage 1.0 (TID 4) in 4429 ms on localhost (executor driver) (3/8)
2024-04-12 20:05:24 INFO TaskSetManager:54 - Finished task 1.0 in stage 1.0 (TID 2) in 4433 ms on localhost (executor driver) (4/8)
2024-04-12 20:05:24 INFO TaskSetManager:54 - Finished task 4.0 in stage 1.0 (TID 5) in 4431 ms on localhost (executor driver) (5/8)
2024-04-12 20:05:24 INFO Executor:54 - Finished task 2.0 in stage 1.0 (TID 3). 1599 bytes result sent to driver
2024-04-12 20:05:24 INFO TaskSetManager:54 - Finished task 2.0 in stage 1.0 (TID 3) in 4470 ms on localhost (executor driver) (6/8)
2024-04-12 20:05:24 INFO Executor:54 - Finished task 0.0 in stage 1.0 (TID 1). 1556 bytes result sent to driver
2024-04-12 20:05:24 INFO TaskSetManager:54 - Finished task 0.0 in stage 1.0 (TID 1) in 4519 ms on localhost (executor driver) (7/8)
2024-04-12 20:05:24 INFO Executor:54 - Finished task 5.0 in stage 1.0 (TID 6). 1556 bytes result sent to driver
2024-04-12 20:05:24 INFO TaskSetManager:54 - Finished task 5.0 in stage 1.0 (TID 6) in 4520 ms on localhost (executor driver) (8/8)
2024-04-12 20:05:24 INFO TaskSchedulerImpl:54 - Removed TaskSet 1.0, whose tasks have all completed, from pool
2024-04-12 20:05:24 INFO DAGScheduler:54 - ResultStage 1 (csv at kmeans.scala:55) finished in 4.546 s
2024-04-12 20:05:24 INFO DAGScheduler:54 - Job 1 finished: csv at kmeans.scala:55, took 4.559625 s
2024-04-12 20:05:24 INFO FileSourceStrategy:54 - Pruning directories with:
2024-04-12 20:05:24 INFO FileSourceStrategy:54 - Post-Scan Filters:
2024-04-12 20:05:24 INFO FileSourceStrategy:54 - Output Data Schema: struct<_c0: double, _c1: double, _c2: double, _c3: double, _c4: double ... 8 more fields>
2024-04-12 20:05:24 INFO FileSourceScanExec:54 - Pushed Filters:
2024-04-12 20:05:24 INFO CodeGenerator:54 - Code generated in 135.928583 ms
2024-04-12 20:05:24 INFO MemoryStore:54 - Block broadcast_4 stored as values in memory (estimated size 277.7 KB, free 365.7 MB)
2024-04-12 20:05:24 INFO MemoryStore:54 - Block broadcast_4_piece0 stored as bytes in memory (estimated size 23.5 KB, free 365.7 MB)
2024-04-12 20:05:24 INFO BlockManagerInfo:54 - Added broadcast_4_piece0 in memory on 10.0.0.154:53058 (size: 23.5 KB, free: 366.2 MB)
2024-04-12 20:05:24 INFO SparkContext:54 - Created broadcast 4 from rdd at KMeans.scala:308
2024-04-12 20:05:24 INFO FileSourceScanExec:54 - Planning scan with bin packing, max size: 57073876 bytes, open cost is considered as scanning 4194304 bytes.
2024-04-12 20:05:24 INFO Instrumentation:54 - KMeans-kmeans_da8d4b500576-379407034-1: training: numPartitions=8 storageLevel=StorageLevel(disk, memory, deserialized, 1 replicas)
2024-04-12 20:05:25 INFO Instrumentation:54 - KMeans-kmeans_da8d4b500576-379407034-1: {"k":10,"seed":1}
2024-04-12 20:05:25 INFO SparkContext:54 - Starting job: takeSample at KMeans.scala:354
2024-04-12 20:05:25 INFO DAGScheduler:54 - Got job 2 (takeSample at KMeans.scala:354) with 8 output partitions
2024-04-12 20:05:25 INFO DAGScheduler:54 - Final stage: ResultStage 2 (takeSample at KMeans.scala:354)
2024-04-12 20:05:25 INFO DAGScheduler:54 - Parents of final stage: List()
2024-04-12 20:05:25 INFO DAGScheduler:54 - Missing parents: List()
2024-04-12 20:05:25 INFO DAGScheduler:54 - Submitting ResultStage 2 (MapPartitionsRDD[16] at map at KMeans.scala:224), which has no missing parents
2024-04-12 20:05:25 INFO MemoryStore:54 - Block broadcast_5 stored as values in memory (estimated size 30.4 KB, free 365.7 MB)
2024-04-12 20:05:25 INFO MemoryStore:54 - Block broadcast_5_piece0 stored as bytes in memory (estimated size 13.7 KB, free 365.6 MB)
2024-04-12 20:05:25 INFO BlockManagerInfo:54 - Added broadcast_5_piece0 in memory on 10.0.0.154:53058 (size: 13.7 KB, free: 366.2 MB)
2024-04-12 20:05:25 INFO SparkContext:54 - Created broadcast 5 from broadcast at DAGScheduler.scala:1039
2024-04-12 20:05:25 INFO DAGScheduler:54 - Submitting 8 missing tasks from ResultStage 2 (MapPartitionsRDD[16] at map at KMeans.scala:224) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7))
2024-04-12 20:05:25 INFO TaskSchedulerImpl:54 - Adding task set 2.0 with 8 tasks
2024-04-12 20:05:25 INFO TaskSetManager:54 - Starting task 0.0 in stage 2.0 (TID 9, localhost, executor driver, partition 0, PROCESS_LOCAL, 8663 bytes)
2024-04-12 20:05:25 INFO TaskSetManager:54 - Starting task 1.0 in stage 2.0 (TID 10, localhost, executor driver, partition 1, PROCESS_LOCAL, 8663 bytes)
2024-04-12 20:05:25 INFO TaskSetManager:54 - Starting task 2.0 in stage 2.0 (TID 11, localhost, executor driver, partition 2, PROCESS_LOCAL, 8663 bytes)
2024-04-12 20:05:25 INFO TaskSetManager:54 - Starting task 3.0 in stage 2.0 (TID 12, localhost, executor driver, partition 3, PROCESS_LOCAL, 8663 bytes)
2024-04-12 20:05:25 INFO TaskSetManager:54 - Starting task 4.0 in stage 2.0 (TID 13, localhost, executor driver, partition 4, PROCESS_LOCAL, 8663 bytes)
2024-04-12 20:05:25 INFO TaskSetManager:54 - Starting task 5.0 in stage 2.0 (TID 14, localhost, executor driver, partition 5, PROCESS_LOCAL, 8663 bytes)
2024-04-12 20:05:25 INFO TaskSetManager:54 - Starting task 6.0 in stage 2.0 (TID 15, localhost, executor driver, partition 6, PROCESS_LOCAL, 8663 bytes)
2024-04-12 20:05:25 INFO TaskSetManager:54 - Starting task 7.0 in stage 2.0 (TID 16, localhost, executor driver, partition 7, PROCESS_LOCAL, 8663 bytes)
2024-04-12 20:05:25 INFO Executor:54 - Running task 7.0 in stage 2.0 (TID 16)
2024-04-12 20:05:25 INFO Executor:54 - Running task 6.0 in stage 2.0 (TID 15)
2024-04-12 20:05:25 INFO Executor:54 - Running task 3.0 in stage 2.0 (TID 12)
2024-04-12 20:05:25 INFO Executor:54 - Running task 5.0 in stage 2.0 (TID 14)
2024-04-12 20:05:25 INFO Executor:54 - Running task 0.0 in stage 2.0 (TID 9)
2024-04-12 20:05:25 INFO Executor:54 - Running task 2.0 in stage 2.0 (TID 11)
2024-04-12 20:05:25 INFO Executor:54 - Running task 1.0 in stage 2.0 (TID 10)
2024-04-12 20:05:25 INFO Executor:54 - Running task 4.0 in stage 2.0 (TID 13)
2024-04-12 20:05:25 INFO CodeGenerator:54 - Code generated in 60.372666 ms
2024-04-12 20:05:25 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 228295504-285369380, partition values: [empty row]
2024-04-12 20:05:25 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 114147752-171221628, partition values: [empty row]
2024-04-12 20:05:25 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 0-57073876, partition values: [empty row]
2024-04-12 20:05:25 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 57073876-114147752, partition values: [empty row]
2024-04-12 20:05:25 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 171221628-228295504, partition values: [empty row]
2024-04-12 20:05:25 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 342443256-399517132, partition values: [empty row]
2024-04-12 20:05:25 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 399517132-452396709, partition values: [empty row]
2024-04-12 20:05:25 INFO FileScanRDD:54 - Reading File path: file:///Users/rithickumar/Desktop/Waterloo_MS/Winter_2024/CS_651/bigdata2024w/project-q1-vectors/part-00000, range: 285369380-342443256, partition values: [empty row]
2024-04-12 20:05:25 INFO CodeGenerator:54 - Code generated in 52.271042 ms
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 50
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 59
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 42
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 44
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 38
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 40
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 46
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 35
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 41
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 52
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 56
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 54
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 36
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 48
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 47
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 60
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 34
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 31
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 37
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 58
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 43
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 49
2024-04-12 20:05:27 INFO BlockManagerInfo:54 - Removed broadcast_3_piece0 on 10.0.0.154:53058 in memory (size: 8.7 KB, free: 366.2 MB)
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 53
2024-04-12 20:05:27 INFO BlockManagerInfo:54 - Removed broadcast_2_piece0 on 10.0.0.154:53058 in memory (size: 23.5 KB, free: 366.3 MB)
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 39
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 57
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 55
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 33
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 45
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 51
2024-04-12 20:05:27 INFO ContextCleaner:54 - Cleaned accumulator 32
2024-04-12 20:05:30 INFO MemoryStore:54 - Block rdd_13_7 stored as values in memory (estimated size 28.9 MB, free 337.1 MB)
2024-04-12 20:05:30 INFO BlockManagerInfo:54 - Added rdd_13_7 in memory on 10.0.0.154:53058 (size: 28.9 MB, free: 337.4 MB)
2024-04-12 20:05:30 INFO BlockManager:54 - Found block rdd_13_7 locally
2024-04-12 20:05:31 INFO MemoryStore:54 - Block rdd_13_0 stored as values in memory (estimated size 31.1 MB, free 241.7 MB)
2024-04-12 20:05:31 INFO MemoryStore:54 - Block rdd_13_5 stored as values in memory (estimated size 31.1 MB, free 241.7 MB)
2024-04-12 20:05:31 INFO MemoryStore:54 - Block rdd_13_4 stored as values in memory (estimated size 31.1 MB, free 241.7 MB)
2024-04-12 20:05:31 INFO BlockManagerInfo:54 - Added rdd_13_0 in memory on 10.0.0.154:53058 (size: 31.1 MB, free: 306.3 MB)
2024-04-12 20:05:31 INFO BlockManagerInfo:54 - Added rdd_13_5 in memory on 10.0.0.154:53058 (size: 31.1 MB, free: 275.2 MB)
2024-04-12 20:05:31 INFO MemoryStore:54 - Block rdd_13_2 stored as values in memory (estimated size 31.1 MB, free 210.6 MB)
2024-04-12 20:05:31 INFO BlockManagerInfo:54 - Added rdd_13_4 in memory on 10.0.0.154:53058 (size: 31.1 MB, free: 244.0 MB)
2024-04-12 20:05:31 INFO MemoryStore:54 - Block rdd_14_7 stored as values in memory (estimated size 2038.2 KB, free 210.6 MB)
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_13_5 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_13_0 locally
2024-04-12 20:05:31 INFO MemoryStore:54 - Block rdd_13_1 stored as values in memory (estimated size 31.1 MB, free 179.5 MB)
2024-04-12 20:05:31 INFO MemoryStore:54 - Block rdd_13_6 stored as values in memory (estimated size 31.1 MB, free 148.3 MB)
2024-04-12 20:05:31 INFO BlockManagerInfo:54 - Added rdd_13_2 in memory on 10.0.0.154:53058 (size: 31.1 MB, free: 212.9 MB)
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_13_4 locally
2024-04-12 20:05:31 INFO BlockManagerInfo:54 - Added rdd_14_7 in memory on 10.0.0.154:53058 (size: 2038.2 KB, free: 210.9 MB)
2024-04-12 20:05:31 INFO BlockManagerInfo:54 - Added rdd_13_1 in memory on 10.0.0.154:53058 (size: 31.1 MB, free: 179.8 MB)
2024-04-12 20:05:31 INFO MemoryStore:54 - Block rdd_13_3 stored as values in memory (estimated size 31.1 MB, free 117.2 MB)
2024-04-12 20:05:31 INFO BlockManagerInfo:54 - Added rdd_13_6 in memory on 10.0.0.154:53058 (size: 31.1 MB, free: 148.6 MB)
2024-04-12 20:05:31 INFO BlockManagerInfo:54 - Added rdd_13_3 in memory on 10.0.0.154:53058 (size: 31.1 MB, free: 117.5 MB)
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_13_2 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_13_1 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_13_3 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_13_6 locally
2024-04-12 20:05:31 INFO MemoryStore:54 - Block rdd_14_0 stored as values in memory (estimated size 2.1 MB, free 108.6 MB)
2024-04-12 20:05:31 INFO MemoryStore:54 - Block rdd_14_4 stored as values in memory (estimated size 2.1 MB, free 106.5 MB)
2024-04-12 20:05:31 INFO MemoryStore:54 - Block rdd_14_5 stored as values in memory (estimated size 2.1 MB, free 108.6 MB)
2024-04-12 20:05:31 INFO MemoryStore:54 - Block rdd_14_3 stored as values in memory (estimated size 2.1 MB, free 106.5 MB)
2024-04-12 20:05:31 INFO MemoryStore:54 - Block rdd_14_2 stored as values in memory (estimated size 2.1 MB, free 106.5 MB)
2024-04-12 20:05:31 INFO BlockManagerInfo:54 - Added rdd_14_0 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 115.4 MB)
2024-04-12 20:05:31 INFO BlockManagerInfo:54 - Added rdd_14_4 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 113.2 MB)
2024-04-12 20:05:31 INFO MemoryStore:54 - Block rdd_14_1 stored as values in memory (estimated size 2.1 MB, free 104.3 MB)
2024-04-12 20:05:31 INFO MemoryStore:54 - Block rdd_14_6 stored as values in memory (estimated size 2.1 MB, free 102.2 MB)
2024-04-12 20:05:31 INFO BlockManagerInfo:54 - Added rdd_14_3 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 111.1 MB)
2024-04-12 20:05:31 INFO BlockManagerInfo:54 - Added rdd_14_5 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 108.9 MB)
2024-04-12 20:05:31 INFO BlockManagerInfo:54 - Added rdd_14_2 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 106.8 MB)
2024-04-12 20:05:31 INFO BlockManagerInfo:54 - Added rdd_14_1 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 104.6 MB)
2024-04-12 20:05:31 INFO BlockManagerInfo:54 - Added rdd_14_6 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 102.5 MB)
2024-04-12 20:05:31 INFO Executor:54 - Finished task 4.0 in stage 2.0 (TID 13). 1518 bytes result sent to driver
2024-04-12 20:05:31 INFO Executor:54 - Finished task 5.0 in stage 2.0 (TID 14). 1475 bytes result sent to driver
2024-04-12 20:05:31 INFO Executor:54 - Finished task 6.0 in stage 2.0 (TID 15). 1518 bytes result sent to driver
2024-04-12 20:05:31 INFO Executor:54 - Finished task 1.0 in stage 2.0 (TID 10). 1475 bytes result sent to driver
2024-04-12 20:05:31 INFO Executor:54 - Finished task 2.0 in stage 2.0 (TID 11). 1475 bytes result sent to driver
2024-04-12 20:05:31 INFO Executor:54 - Finished task 7.0 in stage 2.0 (TID 16). 1518 bytes result sent to driver
2024-04-12 20:05:31 INFO Executor:54 - Finished task 3.0 in stage 2.0 (TID 12). 1475 bytes result sent to driver
2024-04-12 20:05:31 INFO Executor:54 - Finished task 0.0 in stage 2.0 (TID 9). 1475 bytes result sent to driver
2024-04-12 20:05:31 INFO TaskSetManager:54 - Finished task 4.0 in stage 2.0 (TID 13) in 6421 ms on localhost (executor driver) (1/8)
2024-04-12 20:05:31 INFO TaskSetManager:54 - Finished task 7.0 in stage 2.0 (TID 16) in 6421 ms on localhost (executor driver) (2/8)
2024-04-12 20:05:31 INFO TaskSetManager:54 - Finished task 2.0 in stage 2.0 (TID 11) in 6422 ms on localhost (executor driver) (3/8)
2024-04-12 20:05:31 INFO TaskSetManager:54 - Finished task 6.0 in stage 2.0 (TID 15) in 6421 ms on localhost (executor driver) (4/8)
2024-04-12 20:05:31 INFO TaskSetManager:54 - Finished task 1.0 in stage 2.0 (TID 10) in 6427 ms on localhost (executor driver) (5/8)
2024-04-12 20:05:31 INFO TaskSetManager:54 - Finished task 5.0 in stage 2.0 (TID 14) in 6423 ms on localhost (executor driver) (6/8)
2024-04-12 20:05:31 INFO TaskSetManager:54 - Finished task 3.0 in stage 2.0 (TID 12) in 6426 ms on localhost (executor driver) (7/8)
2024-04-12 20:05:31 INFO TaskSetManager:54 - Finished task 0.0 in stage 2.0 (TID 9) in 6436 ms on localhost (executor driver) (8/8)
2024-04-12 20:05:31 INFO TaskSchedulerImpl:54 - Removed TaskSet 2.0, whose tasks have all completed, from pool
2024-04-12 20:05:31 INFO DAGScheduler:54 - ResultStage 2 (takeSample at KMeans.scala:354) finished in 6.531 s
2024-04-12 20:05:31 INFO DAGScheduler:54 - Job 2 finished: takeSample at KMeans.scala:354, took 6.558308 s
2024-04-12 20:05:31 INFO SparkContext:54 - Starting job: takeSample at KMeans.scala:354
2024-04-12 20:05:31 INFO DAGScheduler:54 - Got job 3 (takeSample at KMeans.scala:354) with 8 output partitions
2024-04-12 20:05:31 INFO DAGScheduler:54 - Final stage: ResultStage 3 (takeSample at KMeans.scala:354)
2024-04-12 20:05:31 INFO DAGScheduler:54 - Parents of final stage: List()
2024-04-12 20:05:31 INFO DAGScheduler:54 - Missing parents: List()
2024-04-12 20:05:31 INFO DAGScheduler:54 - Submitting ResultStage 3 (PartitionwiseSampledRDD[18] at takeSample at KMeans.scala:354), which has no missing parents
2024-04-12 20:05:31 INFO MemoryStore:54 - Block broadcast_6 stored as values in memory (estimated size 31.2 KB, free 102.1 MB)
2024-04-12 20:05:31 INFO MemoryStore:54 - Block broadcast_6_piece0 stored as bytes in memory (estimated size 14.1 KB, free 102.1 MB)
2024-04-12 20:05:31 INFO BlockManagerInfo:54 - Added broadcast_6_piece0 in memory on 10.0.0.154:53058 (size: 14.1 KB, free: 102.5 MB)
2024-04-12 20:05:31 INFO SparkContext:54 - Created broadcast 6 from broadcast at DAGScheduler.scala:1039
2024-04-12 20:05:31 INFO DAGScheduler:54 - Submitting 8 missing tasks from ResultStage 3 (PartitionwiseSampledRDD[18] at takeSample at KMeans.scala:354) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7))
2024-04-12 20:05:31 INFO TaskSchedulerImpl:54 - Adding task set 3.0 with 8 tasks
2024-04-12 20:05:31 INFO TaskSetManager:54 - Starting task 0.0 in stage 3.0 (TID 17, localhost, executor driver, partition 0, PROCESS_LOCAL, 8772 bytes)
2024-04-12 20:05:31 INFO TaskSetManager:54 - Starting task 1.0 in stage 3.0 (TID 18, localhost, executor driver, partition 1, PROCESS_LOCAL, 8772 bytes)
2024-04-12 20:05:31 INFO TaskSetManager:54 - Starting task 2.0 in stage 3.0 (TID 19, localhost, executor driver, partition 2, PROCESS_LOCAL, 8772 bytes)
2024-04-12 20:05:31 INFO TaskSetManager:54 - Starting task 3.0 in stage 3.0 (TID 20, localhost, executor driver, partition 3, PROCESS_LOCAL, 8772 bytes)
2024-04-12 20:05:31 INFO TaskSetManager:54 - Starting task 4.0 in stage 3.0 (TID 21, localhost, executor driver, partition 4, PROCESS_LOCAL, 8772 bytes)
2024-04-12 20:05:31 INFO TaskSetManager:54 - Starting task 5.0 in stage 3.0 (TID 22, localhost, executor driver, partition 5, PROCESS_LOCAL, 8772 bytes)
2024-04-12 20:05:31 INFO TaskSetManager:54 - Starting task 6.0 in stage 3.0 (TID 23, localhost, executor driver, partition 6, PROCESS_LOCAL, 8772 bytes)
2024-04-12 20:05:31 INFO TaskSetManager:54 - Starting task 7.0 in stage 3.0 (TID 24, localhost, executor driver, partition 7, PROCESS_LOCAL, 8772 bytes)
2024-04-12 20:05:31 INFO Executor:54 - Running task 0.0 in stage 3.0 (TID 17)
2024-04-12 20:05:31 INFO Executor:54 - Running task 7.0 in stage 3.0 (TID 24)
2024-04-12 20:05:31 INFO Executor:54 - Running task 2.0 in stage 3.0 (TID 19)
2024-04-12 20:05:31 INFO Executor:54 - Running task 6.0 in stage 3.0 (TID 23)
2024-04-12 20:05:31 INFO Executor:54 - Running task 5.0 in stage 3.0 (TID 22)
2024-04-12 20:05:31 INFO Executor:54 - Running task 1.0 in stage 3.0 (TID 18)
2024-04-12 20:05:31 INFO Executor:54 - Running task 4.0 in stage 3.0 (TID 21)
2024-04-12 20:05:31 INFO Executor:54 - Running task 3.0 in stage 3.0 (TID 20)
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_13_6 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_13_5 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_13_0 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_13_4 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_13_7 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_13_3 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_13_1 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_13_2 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_14_4 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_14_0 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_14_5 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_14_6 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_14_1 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_14_3 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_14_2 locally
2024-04-12 20:05:31 INFO BlockManager:54 - Found block rdd_14_7 locally
2024-04-12 20:05:31 INFO Executor:54 - Finished task 3.0 in stage 3.0 (TID 20). 1997 bytes result sent to driver
2024-04-12 20:05:31 INFO Executor:54 - Finished task 6.0 in stage 3.0 (TID 23). 2107 bytes result sent to driver
2024-04-12 20:05:31 INFO Executor:54 - Finished task 1.0 in stage 3.0 (TID 18). 1887 bytes result sent to driver
2024-04-12 20:05:31 INFO Executor:54 - Finished task 5.0 in stage 3.0 (TID 22). 1997 bytes result sent to driver
2024-04-12 20:05:31 INFO Executor:54 - Finished task 7.0 in stage 3.0 (TID 24). 1777 bytes result sent to driver
2024-04-12 20:05:31 INFO Executor:54 - Finished task 2.0 in stage 3.0 (TID 19). 2217 bytes result sent to driver
2024-04-12 20:05:31 INFO Executor:54 - Finished task 0.0 in stage 3.0 (TID 17). 2107 bytes result sent to driver
2024-04-12 20:05:32 INFO Executor:54 - Finished task 4.0 in stage 3.0 (TID 21). 2396 bytes result sent to driver
2024-04-12 20:05:32 INFO TaskSetManager:54 - Finished task 6.0 in stage 3.0 (TID 23) in 284 ms on localhost (executor driver) (1/8)
2024-04-12 20:05:32 INFO TaskSetManager:54 - Finished task 1.0 in stage 3.0 (TID 18) in 286 ms on localhost (executor driver) (2/8)
2024-04-12 20:05:32 INFO TaskSetManager:54 - Finished task 3.0 in stage 3.0 (TID 20) in 286 ms on localhost (executor driver) (3/8)
2024-04-12 20:05:32 INFO TaskSetManager:54 - Finished task 5.0 in stage 3.0 (TID 22) in 286 ms on localhost (executor driver) (4/8)
2024-04-12 20:05:32 INFO TaskSetManager:54 - Finished task 0.0 in stage 3.0 (TID 17) in 291 ms on localhost (executor driver) (5/8)
2024-04-12 20:05:32 INFO TaskSetManager:54 - Finished task 7.0 in stage 3.0 (TID 24) in 284 ms on localhost (executor driver) (6/8)
2024-04-12 20:05:32 INFO TaskSetManager:54 - Finished task 2.0 in stage 3.0 (TID 19) in 287 ms on localhost (executor driver) (7/8)
2024-04-12 20:05:32 INFO TaskSetManager:54 - Finished task 4.0 in stage 3.0 (TID 21) in 286 ms on localhost (executor driver) (8/8)
2024-04-12 20:05:32 INFO TaskSchedulerImpl:54 - Removed TaskSet 3.0, whose tasks have all completed, from pool
2024-04-12 20:05:32 INFO DAGScheduler:54 - ResultStage 3 (takeSample at KMeans.scala:354) finished in 0.316 s
2024-04-12 20:05:32 INFO DAGScheduler:54 - Job 3 finished: takeSample at KMeans.scala:354, took 0.342407 s
2024-04-12 20:05:32 INFO MemoryStore:54 - Block broadcast_7 stored as values in memory (estimated size 200.0 B, free 102.1 MB)
2024-04-12 20:05:32 INFO MemoryStore:54 - Block broadcast_7_piece0 stored as bytes in memory (estimated size 412.0 B, free 102.1 MB)
2024-04-12 20:05:32 INFO BlockManagerInfo:54 - Added broadcast_7_piece0 in memory on 10.0.0.154:53058 (size: 412.0 B, free: 102.5 MB)
2024-04-12 20:05:32 INFO SparkContext:54 - Created broadcast 7 from broadcast at KMeans.scala:368
2024-04-12 20:05:32 INFO SparkContext:54 - Starting job: sum at KMeans.scala:374
2024-04-12 20:05:32 INFO DAGScheduler:54 - Got job 4 (sum at KMeans.scala:374) with 8 output partitions
2024-04-12 20:05:32 INFO DAGScheduler:54 - Final stage: ResultStage 4 (sum at KMeans.scala:374)
2024-04-12 20:05:32 INFO DAGScheduler:54 - Parents of final stage: List()
2024-04-12 20:05:32 INFO DAGScheduler:54 - Missing parents: List()
2024-04-12 20:05:32 INFO DAGScheduler:54 - Submitting ResultStage 4 (MapPartitionsRDD[20] at map at KMeans.scala:371), which has no missing parents
2024-04-12 20:05:32 INFO MemoryStore:54 - Block broadcast_8 stored as values in memory (estimated size 31.3 KB, free 102.1 MB)
2024-04-12 20:05:32 INFO MemoryStore:54 - Block broadcast_8_piece0 stored as bytes in memory (estimated size 14.1 KB, free 102.1 MB)
2024-04-12 20:05:32 INFO BlockManagerInfo:54 - Added broadcast_8_piece0 in memory on 10.0.0.154:53058 (size: 14.1 KB, free: 102.4 MB)
2024-04-12 20:05:32 INFO SparkContext:54 - Created broadcast 8 from broadcast at DAGScheduler.scala:1039
2024-04-12 20:05:32 INFO DAGScheduler:54 - Submitting 8 missing tasks from ResultStage 4 (MapPartitionsRDD[20] at map at KMeans.scala:371) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7))
2024-04-12 20:05:32 INFO TaskSchedulerImpl:54 - Adding task set 4.0 with 8 tasks
2024-04-12 20:05:32 INFO TaskSetManager:54 - Starting task 0.0 in stage 4.0 (TID 25, localhost, executor driver, partition 0, PROCESS_LOCAL, 8695 bytes)
2024-04-12 20:05:32 INFO TaskSetManager:54 - Starting task 1.0 in stage 4.0 (TID 26, localhost, executor driver, partition 1, PROCESS_LOCAL, 8695 bytes)
2024-04-12 20:05:32 INFO TaskSetManager:54 - Starting task 2.0 in stage 4.0 (TID 27, localhost, executor driver, partition 2, PROCESS_LOCAL, 8695 bytes)
2024-04-12 20:05:32 INFO TaskSetManager:54 - Starting task 3.0 in stage 4.0 (TID 28, localhost, executor driver, partition 3, PROCESS_LOCAL, 8695 bytes)
2024-04-12 20:05:32 INFO TaskSetManager:54 - Starting task 4.0 in stage 4.0 (TID 29, localhost, executor driver, partition 4, PROCESS_LOCAL, 8695 bytes)
2024-04-12 20:05:32 INFO TaskSetManager:54 - Starting task 5.0 in stage 4.0 (TID 30, localhost, executor driver, partition 5, PROCESS_LOCAL, 8695 bytes)
2024-04-12 20:05:32 INFO TaskSetManager:54 - Starting task 6.0 in stage 4.0 (TID 31, localhost, executor driver, partition 6, PROCESS_LOCAL, 8695 bytes)
2024-04-12 20:05:32 INFO TaskSetManager:54 - Starting task 7.0 in stage 4.0 (TID 32, localhost, executor driver, partition 7, PROCESS_LOCAL, 8695 bytes)
2024-04-12 20:05:32 INFO Executor:54 - Running task 5.0 in stage 4.0 (TID 30)
2024-04-12 20:05:32 INFO Executor:54 - Running task 4.0 in stage 4.0 (TID 29)
2024-04-12 20:05:32 INFO Executor:54 - Running task 0.0 in stage 4.0 (TID 25)
2024-04-12 20:05:32 INFO Executor:54 - Running task 2.0 in stage 4.0 (TID 27)
2024-04-12 20:05:32 INFO Executor:54 - Running task 7.0 in stage 4.0 (TID 32)
2024-04-12 20:05:32 INFO Executor:54 - Running task 3.0 in stage 4.0 (TID 28)
2024-04-12 20:05:32 INFO Executor:54 - Running task 1.0 in stage 4.0 (TID 26)
2024-04-12 20:05:32 INFO Executor:54 - Running task 6.0 in stage 4.0 (TID 31)
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_13_5 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_13_7 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_13_1 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_13_2 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_13_4 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_13_6 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_13_0 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_13_3 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_14_6 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_14_4 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_14_2 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_14_1 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_13_4 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_14_7 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_14_5 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_13_7 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_14_4 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_13_2 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_14_2 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_13_1 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_13_6 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_14_3 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_14_0 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_13_3 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_14_6 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_14_1 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_14_7 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_13_5 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_14_3 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_13_0 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_14_5 locally
2024-04-12 20:05:32 INFO BlockManager:54 - Found block rdd_14_0 locally
2024-04-12 20:05:32 WARN BLAS:61 - Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
2024-04-12 20:05:32 WARN BLAS:61 - Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 106
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 96
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 108
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 111
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 92
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 107
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 93
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 101
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 110
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 100
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 94
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 115
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 95
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 109
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 91
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 102
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 99
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 104
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 114
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 113
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 112
2024-04-12 20:05:32 INFO BlockManagerInfo:54 - Removed broadcast_6_piece0 on 10.0.0.154:53058 in memory (size: 14.1 KB, free: 102.5 MB)
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 105
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 98
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 97
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 103
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 83
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 74
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 88
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 85
2024-04-12 20:05:32 INFO ContextCleaner:54 - Cleaned accumulator 67
2024-04-12 20:05:33 INFO MemoryStore:54 - Block rdd_20_2 stored as values in memory (estimated size 2.1 MB, free 87.1 MB)
2024-04-12 20:05:33 INFO MemoryStore:54 - Block rdd_20_4 stored as values in memory (estimated size 2.1 MB, free 87.1 MB)
2024-04-12 20:05:33 INFO MemoryStore:54 - Block rdd_20_0 stored as values in memory (estimated size 2.1 MB, free 87.1 MB)
2024-04-12 20:05:33 INFO MemoryStore:54 - Block rdd_20_1 stored as values in memory (estimated size 2.1 MB, free 87.1 MB)
2024-04-12 20:05:33 INFO MemoryStore:54 - Block rdd_20_5 stored as values in memory (estimated size 2.1 MB, free 87.1 MB)
2024-04-12 20:05:33 INFO MemoryStore:54 - Block rdd_20_3 stored as values in memory (estimated size 2.1 MB, free 87.1 MB)
2024-04-12 20:05:33 INFO MemoryStore:54 - Block rdd_20_6 stored as values in memory (estimated size 2.1 MB, free 87.1 MB)
2024-04-12 20:05:33 INFO BlockManagerInfo:54 - Removed broadcast_5_piece0 on 10.0.0.154:53058 in memory (size: 13.7 KB, free: 102.5 MB)
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 86
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 66
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 90
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 81
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 78
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 72
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 71
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 89
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 70
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 76
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 75
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 73
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 80
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 79
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 84
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 68
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 69
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 82
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 77
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 87
2024-04-12 20:05:33 INFO MemoryStore:54 - Block rdd_20_7 stored as values in memory (estimated size 2038.2 KB, free 85.2 MB)
2024-04-12 20:05:33 INFO BlockManagerInfo:54 - Added rdd_20_0 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 100.3 MB)
2024-04-12 20:05:33 INFO BlockManagerInfo:54 - Added rdd_20_4 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 98.2 MB)
2024-04-12 20:05:33 INFO BlockManagerInfo:54 - Added rdd_20_6 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 96.0 MB)
2024-04-12 20:05:33 INFO BlockManagerInfo:54 - Added rdd_20_5 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 93.9 MB)
2024-04-12 20:05:33 INFO BlockManagerInfo:54 - Added rdd_20_3 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 91.7 MB)
2024-04-12 20:05:33 INFO BlockManagerInfo:54 - Added rdd_20_1 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 89.6 MB)
2024-04-12 20:05:33 INFO BlockManagerInfo:54 - Added rdd_20_2 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 87.4 MB)
2024-04-12 20:05:33 INFO Executor:54 - Finished task 6.0 in stage 4.0 (TID 31). 1520 bytes result sent to driver
2024-04-12 20:05:33 INFO Executor:54 - Finished task 4.0 in stage 4.0 (TID 29). 1520 bytes result sent to driver
2024-04-12 20:05:33 INFO Executor:54 - Finished task 0.0 in stage 4.0 (TID 25). 1520 bytes result sent to driver
2024-04-12 20:05:33 INFO BlockManagerInfo:54 - Added rdd_20_7 in memory on 10.0.0.154:53058 (size: 2038.2 KB, free: 85.5 MB)
2024-04-12 20:05:33 INFO Executor:54 - Finished task 5.0 in stage 4.0 (TID 30). 1520 bytes result sent to driver
2024-04-12 20:05:33 INFO Executor:54 - Finished task 3.0 in stage 4.0 (TID 28). 1477 bytes result sent to driver
2024-04-12 20:05:33 INFO TaskSetManager:54 - Finished task 4.0 in stage 4.0 (TID 29) in 962 ms on localhost (executor driver) (1/8)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Finished task 0.0 in stage 4.0 (TID 25) in 967 ms on localhost (executor driver) (2/8)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Finished task 5.0 in stage 4.0 (TID 30) in 965 ms on localhost (executor driver) (3/8)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Finished task 6.0 in stage 4.0 (TID 31) in 963 ms on localhost (executor driver) (4/8)
2024-04-12 20:05:33 INFO Executor:54 - Finished task 7.0 in stage 4.0 (TID 32). 1520 bytes result sent to driver
2024-04-12 20:05:33 INFO Executor:54 - Finished task 2.0 in stage 4.0 (TID 27). 1520 bytes result sent to driver
2024-04-12 20:05:33 INFO TaskSetManager:54 - Finished task 3.0 in stage 4.0 (TID 28) in 973 ms on localhost (executor driver) (5/8)
2024-04-12 20:05:33 INFO Executor:54 - Finished task 1.0 in stage 4.0 (TID 26). 1520 bytes result sent to driver
2024-04-12 20:05:33 INFO TaskSetManager:54 - Finished task 7.0 in stage 4.0 (TID 32) in 974 ms on localhost (executor driver) (6/8)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Finished task 2.0 in stage 4.0 (TID 27) in 977 ms on localhost (executor driver) (7/8)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Finished task 1.0 in stage 4.0 (TID 26) in 979 ms on localhost (executor driver) (8/8)
2024-04-12 20:05:33 INFO TaskSchedulerImpl:54 - Removed TaskSet 4.0, whose tasks have all completed, from pool
2024-04-12 20:05:33 INFO DAGScheduler:54 - ResultStage 4 (sum at KMeans.scala:374) finished in 1.017 s
2024-04-12 20:05:33 INFO DAGScheduler:54 - Job 4 finished: sum at KMeans.scala:374, took 1.035188 s
2024-04-12 20:05:33 INFO MapPartitionsRDD:54 - Removing RDD 17 from persistence list
2024-04-12 20:05:33 INFO BlockManager:54 - Removing RDD 17
2024-04-12 20:05:33 INFO SparkContext:54 - Starting job: collect at KMeans.scala:382
2024-04-12 20:05:33 INFO DAGScheduler:54 - Got job 5 (collect at KMeans.scala:382) with 8 output partitions
2024-04-12 20:05:33 INFO DAGScheduler:54 - Final stage: ResultStage 5 (collect at KMeans.scala:382)
2024-04-12 20:05:33 INFO DAGScheduler:54 - Parents of final stage: List()
2024-04-12 20:05:33 INFO DAGScheduler:54 - Missing parents: List()
2024-04-12 20:05:33 INFO DAGScheduler:54 - Submitting ResultStage 5 (MapPartitionsRDD[22] at mapPartitionsWithIndex at KMeans.scala:379), which has no missing parents
2024-04-12 20:05:33 INFO MemoryStore:54 - Block broadcast_9 stored as values in memory (estimated size 31.9 KB, free 85.1 MB)
2024-04-12 20:05:33 INFO MemoryStore:54 - Block broadcast_9_piece0 stored as bytes in memory (estimated size 14.3 KB, free 85.1 MB)
2024-04-12 20:05:33 INFO BlockManagerInfo:54 - Added broadcast_9_piece0 in memory on 10.0.0.154:53058 (size: 14.3 KB, free: 85.4 MB)
2024-04-12 20:05:33 INFO SparkContext:54 - Created broadcast 9 from broadcast at DAGScheduler.scala:1039
2024-04-12 20:05:33 INFO DAGScheduler:54 - Submitting 8 missing tasks from ResultStage 5 (MapPartitionsRDD[22] at mapPartitionsWithIndex at KMeans.scala:379) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7))
2024-04-12 20:05:33 INFO TaskSchedulerImpl:54 - Adding task set 5.0 with 8 tasks
2024-04-12 20:05:33 INFO TaskSetManager:54 - Starting task 0.0 in stage 5.0 (TID 33, localhost, executor driver, partition 0, PROCESS_LOCAL, 8727 bytes)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Starting task 1.0 in stage 5.0 (TID 34, localhost, executor driver, partition 1, PROCESS_LOCAL, 8727 bytes)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Starting task 2.0 in stage 5.0 (TID 35, localhost, executor driver, partition 2, PROCESS_LOCAL, 8727 bytes)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Starting task 3.0 in stage 5.0 (TID 36, localhost, executor driver, partition 3, PROCESS_LOCAL, 8727 bytes)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Starting task 4.0 in stage 5.0 (TID 37, localhost, executor driver, partition 4, PROCESS_LOCAL, 8727 bytes)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Starting task 5.0 in stage 5.0 (TID 38, localhost, executor driver, partition 5, PROCESS_LOCAL, 8727 bytes)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Starting task 6.0 in stage 5.0 (TID 39, localhost, executor driver, partition 6, PROCESS_LOCAL, 8727 bytes)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Starting task 7.0 in stage 5.0 (TID 40, localhost, executor driver, partition 7, PROCESS_LOCAL, 8727 bytes)
2024-04-12 20:05:33 INFO Executor:54 - Running task 1.0 in stage 5.0 (TID 34)
2024-04-12 20:05:33 INFO Executor:54 - Running task 7.0 in stage 5.0 (TID 40)
2024-04-12 20:05:33 INFO Executor:54 - Running task 4.0 in stage 5.0 (TID 37)
2024-04-12 20:05:33 INFO Executor:54 - Running task 3.0 in stage 5.0 (TID 36)
2024-04-12 20:05:33 INFO Executor:54 - Running task 2.0 in stage 5.0 (TID 35)
2024-04-12 20:05:33 INFO Executor:54 - Running task 5.0 in stage 5.0 (TID 38)
2024-04-12 20:05:33 INFO Executor:54 - Running task 0.0 in stage 5.0 (TID 33)
2024-04-12 20:05:33 INFO Executor:54 - Running task 6.0 in stage 5.0 (TID 39)
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_13_7 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_14_7 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_13_4 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_20_7 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_14_4 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_20_4 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_13_5 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_13_3 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_13_1 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_13_0 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_14_0 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_20_0 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_13_2 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_14_3 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_14_5 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_13_6 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_20_3 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_14_1 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_14_2 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_14_6 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_20_2 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_20_1 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_20_5 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_20_6 locally
2024-04-12 20:05:33 INFO Executor:54 - Finished task 2.0 in stage 5.0 (TID 35). 1930 bytes result sent to driver
2024-04-12 20:05:33 INFO Executor:54 - Finished task 5.0 in stage 5.0 (TID 38). 2260 bytes result sent to driver
2024-04-12 20:05:33 INFO Executor:54 - Finished task 3.0 in stage 5.0 (TID 36). 2107 bytes result sent to driver
2024-04-12 20:05:33 INFO Executor:54 - Finished task 4.0 in stage 5.0 (TID 37). 1887 bytes result sent to driver
2024-04-12 20:05:33 INFO Executor:54 - Finished task 6.0 in stage 5.0 (TID 39). 2107 bytes result sent to driver
2024-04-12 20:05:33 INFO Executor:54 - Finished task 1.0 in stage 5.0 (TID 34). 1469 bytes result sent to driver
2024-04-12 20:05:33 INFO Executor:54 - Finished task 0.0 in stage 5.0 (TID 33). 1887 bytes result sent to driver
2024-04-12 20:05:33 INFO TaskSetManager:54 - Finished task 4.0 in stage 5.0 (TID 37) in 177 ms on localhost (executor driver) (1/8)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Finished task 3.0 in stage 5.0 (TID 36) in 179 ms on localhost (executor driver) (2/8)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Finished task 2.0 in stage 5.0 (TID 35) in 179 ms on localhost (executor driver) (3/8)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Finished task 5.0 in stage 5.0 (TID 38) in 178 ms on localhost (executor driver) (4/8)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Finished task 6.0 in stage 5.0 (TID 39) in 178 ms on localhost (executor driver) (5/8)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Finished task 0.0 in stage 5.0 (TID 33) in 181 ms on localhost (executor driver) (6/8)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Finished task 1.0 in stage 5.0 (TID 34) in 180 ms on localhost (executor driver) (7/8)
2024-04-12 20:05:33 INFO Executor:54 - Finished task 7.0 in stage 5.0 (TID 40). 1777 bytes result sent to driver
2024-04-12 20:05:33 INFO TaskSetManager:54 - Finished task 7.0 in stage 5.0 (TID 40) in 187 ms on localhost (executor driver) (8/8)
2024-04-12 20:05:33 INFO TaskSchedulerImpl:54 - Removed TaskSet 5.0, whose tasks have all completed, from pool
2024-04-12 20:05:33 INFO DAGScheduler:54 - ResultStage 5 (collect at KMeans.scala:382) finished in 0.218 s
2024-04-12 20:05:33 INFO DAGScheduler:54 - Job 5 finished: collect at KMeans.scala:382, took 0.239148 s
2024-04-12 20:05:33 INFO MemoryStore:54 - Block broadcast_10 stored as values in memory (estimated size 2.8 KB, free 85.1 MB)
2024-04-12 20:05:33 INFO MemoryStore:54 - Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.1 KB, free 85.1 MB)
2024-04-12 20:05:33 INFO BlockManagerInfo:54 - Added broadcast_10_piece0 in memory on 10.0.0.154:53058 (size: 2.1 KB, free: 85.4 MB)
2024-04-12 20:05:33 INFO SparkContext:54 - Created broadcast 10 from broadcast at KMeans.scala:368
2024-04-12 20:05:33 INFO SparkContext:54 - Starting job: sum at KMeans.scala:374
2024-04-12 20:05:33 INFO DAGScheduler:54 - Got job 6 (sum at KMeans.scala:374) with 8 output partitions
2024-04-12 20:05:33 INFO DAGScheduler:54 - Final stage: ResultStage 6 (sum at KMeans.scala:374)
2024-04-12 20:05:33 INFO DAGScheduler:54 - Parents of final stage: List()
2024-04-12 20:05:33 INFO DAGScheduler:54 - Missing parents: List()
2024-04-12 20:05:33 INFO DAGScheduler:54 - Submitting ResultStage 6 (MapPartitionsRDD[24] at map at KMeans.scala:371), which has no missing parents
2024-04-12 20:05:33 INFO MemoryStore:54 - Block broadcast_11 stored as values in memory (estimated size 31.5 KB, free 85.1 MB)
2024-04-12 20:05:33 INFO MemoryStore:54 - Block broadcast_11_piece0 stored as bytes in memory (estimated size 14.1 KB, free 85.1 MB)
2024-04-12 20:05:33 INFO BlockManagerInfo:54 - Added broadcast_11_piece0 in memory on 10.0.0.154:53058 (size: 14.1 KB, free: 85.4 MB)
2024-04-12 20:05:33 INFO SparkContext:54 - Created broadcast 11 from broadcast at DAGScheduler.scala:1039
2024-04-12 20:05:33 INFO DAGScheduler:54 - Submitting 8 missing tasks from ResultStage 6 (MapPartitionsRDD[24] at map at KMeans.scala:371) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7))
2024-04-12 20:05:33 INFO TaskSchedulerImpl:54 - Adding task set 6.0 with 8 tasks
2024-04-12 20:05:33 INFO TaskSetManager:54 - Starting task 0.0 in stage 6.0 (TID 41, localhost, executor driver, partition 0, PROCESS_LOCAL, 8727 bytes)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Starting task 1.0 in stage 6.0 (TID 42, localhost, executor driver, partition 1, PROCESS_LOCAL, 8727 bytes)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Starting task 2.0 in stage 6.0 (TID 43, localhost, executor driver, partition 2, PROCESS_LOCAL, 8727 bytes)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Starting task 3.0 in stage 6.0 (TID 44, localhost, executor driver, partition 3, PROCESS_LOCAL, 8727 bytes)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Starting task 4.0 in stage 6.0 (TID 45, localhost, executor driver, partition 4, PROCESS_LOCAL, 8727 bytes)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Starting task 5.0 in stage 6.0 (TID 46, localhost, executor driver, partition 5, PROCESS_LOCAL, 8727 bytes)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Starting task 6.0 in stage 6.0 (TID 47, localhost, executor driver, partition 6, PROCESS_LOCAL, 8727 bytes)
2024-04-12 20:05:33 INFO TaskSetManager:54 - Starting task 7.0 in stage 6.0 (TID 48, localhost, executor driver, partition 7, PROCESS_LOCAL, 8727 bytes)
2024-04-12 20:05:33 INFO Executor:54 - Running task 0.0 in stage 6.0 (TID 41)
2024-04-12 20:05:33 INFO Executor:54 - Running task 2.0 in stage 6.0 (TID 43)
2024-04-12 20:05:33 INFO Executor:54 - Running task 1.0 in stage 6.0 (TID 42)
2024-04-12 20:05:33 INFO Executor:54 - Running task 5.0 in stage 6.0 (TID 46)
2024-04-12 20:05:33 INFO Executor:54 - Running task 7.0 in stage 6.0 (TID 48)
2024-04-12 20:05:33 INFO Executor:54 - Running task 4.0 in stage 6.0 (TID 45)
2024-04-12 20:05:33 INFO Executor:54 - Running task 3.0 in stage 6.0 (TID 44)
2024-04-12 20:05:33 INFO Executor:54 - Running task 6.0 in stage 6.0 (TID 47)
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_13_1 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_13_5 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_13_3 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_13_0 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_14_0 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_20_0 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_13_2 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_14_3 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_14_1 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_20_3 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_14_2 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_14_5 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_20_1 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_20_2 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_13_6 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_20_5 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_13_7 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_13_4 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_14_7 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_14_6 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_14_4 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_20_7 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_20_4 locally
2024-04-12 20:05:33 INFO BlockManager:54 - Found block rdd_20_6 locally
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 151
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 161
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 149
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 150
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 162
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 144
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 164
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 147
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 154
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 160
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 163
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 142
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 153
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 143
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 165
2024-04-12 20:05:33 INFO BlockManagerInfo:54 - Removed broadcast_9_piece0 on 10.0.0.154:53058 in memory (size: 14.3 KB, free: 85.4 MB)
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 158
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 145
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 148
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 155
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 157
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 156
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 141
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 146
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 152
2024-04-12 20:05:33 INFO ContextCleaner:54 - Cleaned accumulator 159
2024-04-12 20:05:34 INFO MemoryStore:54 - Block rdd_24_2 stored as values in memory (estimated size 2.1 MB, free 78.8 MB)
2024-04-12 20:05:34 INFO MemoryStore:54 - Block rdd_24_7 stored as values in memory (estimated size 2038.2 KB, free 78.8 MB)
2024-04-12 20:05:34 INFO MemoryStore:54 - Block rdd_24_0 stored as values in memory (estimated size 2.1 MB, free 78.8 MB)
2024-04-12 20:05:34 INFO BlockManagerInfo:54 - Added rdd_24_0 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 83.3 MB)
2024-04-12 20:05:34 INFO BlockManagerInfo:54 - Added rdd_24_2 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 81.1 MB)
2024-04-12 20:05:34 INFO MemoryStore:54 - Block rdd_24_3 stored as values in memory (estimated size 2.1 MB, free 76.7 MB)
2024-04-12 20:05:34 INFO BlockManagerInfo:54 - Added rdd_24_7 in memory on 10.0.0.154:53058 (size: 2038.2 KB, free: 79.2 MB)
2024-04-12 20:05:34 INFO BlockManagerInfo:54 - Added rdd_24_3 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 77.0 MB)
2024-04-12 20:05:34 INFO MemoryStore:54 - Block rdd_24_1 stored as values in memory (estimated size 2.1 MB, free 70.2 MB)
2024-04-12 20:05:34 INFO MemoryStore:54 - Block rdd_24_4 stored as values in memory (estimated size 2.1 MB, free 70.2 MB)
2024-04-12 20:05:34 INFO MemoryStore:54 - Block rdd_24_6 stored as values in memory (estimated size 2.1 MB, free 70.2 MB)
2024-04-12 20:05:34 INFO BlockManagerInfo:54 - Added rdd_24_1 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 74.9 MB)
2024-04-12 20:05:34 INFO BlockManagerInfo:54 - Added rdd_24_4 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 72.7 MB)
2024-04-12 20:05:34 INFO MemoryStore:54 - Block rdd_24_5 stored as values in memory (estimated size 2.1 MB, free 68.1 MB)
2024-04-12 20:05:34 INFO Executor:54 - Finished task 0.0 in stage 6.0 (TID 41). 1520 bytes result sent to driver
2024-04-12 20:05:34 INFO BlockManagerInfo:54 - Added rdd_24_6 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 70.6 MB)
2024-04-12 20:05:34 INFO Executor:54 - Finished task 7.0 in stage 6.0 (TID 48). 1520 bytes result sent to driver
2024-04-12 20:05:34 INFO Executor:54 - Finished task 2.0 in stage 6.0 (TID 43). 1477 bytes result sent to driver
2024-04-12 20:05:34 INFO BlockManagerInfo:54 - Added rdd_24_5 in memory on 10.0.0.154:53058 (size: 2.1 MB, free: 68.4 MB)
2024-04-12 20:05:34 INFO Executor:54 - Finished task 3.0 in stage 6.0 (TID 44). 1477 bytes result sent to driver
2024-04-12 20:05:34 INFO Executor:54 - Finished task 1.0 in stage 6.0 (TID 42). 1477 bytes result sent to driver
2024-04-12 20:05:34 INFO Executor:54 - Finished task 6.0 in stage 6.0 (TID 47). 1477 bytes result sent to driver
2024-04-12 20:05:34 INFO Executor:54 - Finished task 4.0 in stage 6.0 (TID 45). 1477 bytes result sent to driver
2024-04-12 20:05:34 INFO TaskSetManager:54 - Finished task 3.0 in stage 6.0 (TID 44) in 522 ms on localhost (executor driver) (1/8)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Finished task 0.0 in stage 6.0 (TID 41) in 525 ms on localhost (executor driver) (2/8)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Finished task 2.0 in stage 6.0 (TID 43) in 524 ms on localhost (executor driver) (3/8)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Finished task 7.0 in stage 6.0 (TID 48) in 521 ms on localhost (executor driver) (4/8)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Finished task 1.0 in stage 6.0 (TID 42) in 526 ms on localhost (executor driver) (5/8)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Finished task 6.0 in stage 6.0 (TID 47) in 521 ms on localhost (executor driver) (6/8)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Finished task 4.0 in stage 6.0 (TID 45) in 522 ms on localhost (executor driver) (7/8)
2024-04-12 20:05:34 INFO Executor:54 - Finished task 5.0 in stage 6.0 (TID 46). 1477 bytes result sent to driver
2024-04-12 20:05:34 INFO TaskSetManager:54 - Finished task 5.0 in stage 6.0 (TID 46) in 527 ms on localhost (executor driver) (8/8)
2024-04-12 20:05:34 INFO TaskSchedulerImpl:54 - Removed TaskSet 6.0, whose tasks have all completed, from pool
2024-04-12 20:05:34 INFO DAGScheduler:54 - ResultStage 6 (sum at KMeans.scala:374) finished in 0.549 s
2024-04-12 20:05:34 INFO DAGScheduler:54 - Job 6 finished: sum at KMeans.scala:374, took 0.567544 s
2024-04-12 20:05:34 INFO MapPartitionsRDD:54 - Removing RDD 20 from persistence list
2024-04-12 20:05:34 INFO BlockManager:54 - Removing RDD 20
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 182
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 188
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 175
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 185
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 181
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 176
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 186
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 190
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 167
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 168
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 170
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 169
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 166
2024-04-12 20:05:34 INFO BlockManagerInfo:54 - Removed broadcast_11_piece0 on 10.0.0.154:53058 in memory (size: 14.1 KB, free: 85.5 MB)
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 184
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 174
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 179
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 177
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 180
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 183
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 189
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 172
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 178
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 187
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 171
2024-04-12 20:05:34 INFO ContextCleaner:54 - Cleaned accumulator 173
2024-04-12 20:05:34 INFO SparkContext:54 - Starting job: collect at KMeans.scala:382
2024-04-12 20:05:34 INFO DAGScheduler:54 - Got job 7 (collect at KMeans.scala:382) with 8 output partitions
2024-04-12 20:05:34 INFO DAGScheduler:54 - Final stage: ResultStage 7 (collect at KMeans.scala:382)
2024-04-12 20:05:34 INFO DAGScheduler:54 - Parents of final stage: List()
2024-04-12 20:05:34 INFO DAGScheduler:54 - Missing parents: List()
2024-04-12 20:05:34 INFO DAGScheduler:54 - Submitting ResultStage 7 (MapPartitionsRDD[26] at mapPartitionsWithIndex at KMeans.scala:379), which has no missing parents
2024-04-12 20:05:34 INFO MemoryStore:54 - Block broadcast_12 stored as values in memory (estimated size 32.1 KB, free 85.1 MB)
2024-04-12 20:05:34 INFO MemoryStore:54 - Block broadcast_12_piece0 stored as bytes in memory (estimated size 14.4 KB, free 85.1 MB)
2024-04-12 20:05:34 INFO BlockManagerInfo:54 - Added broadcast_12_piece0 in memory on 10.0.0.154:53058 (size: 14.4 KB, free: 85.4 MB)
2024-04-12 20:05:34 INFO SparkContext:54 - Created broadcast 12 from broadcast at DAGScheduler.scala:1039
2024-04-12 20:05:34 INFO DAGScheduler:54 - Submitting 8 missing tasks from ResultStage 7 (MapPartitionsRDD[26] at mapPartitionsWithIndex at KMeans.scala:379) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7))
2024-04-12 20:05:34 INFO TaskSchedulerImpl:54 - Adding task set 7.0 with 8 tasks
2024-04-12 20:05:34 INFO TaskSetManager:54 - Starting task 0.0 in stage 7.0 (TID 49, localhost, executor driver, partition 0, PROCESS_LOCAL, 8759 bytes)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Starting task 1.0 in stage 7.0 (TID 50, localhost, executor driver, partition 1, PROCESS_LOCAL, 8759 bytes)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Starting task 2.0 in stage 7.0 (TID 51, localhost, executor driver, partition 2, PROCESS_LOCAL, 8759 bytes)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Starting task 3.0 in stage 7.0 (TID 52, localhost, executor driver, partition 3, PROCESS_LOCAL, 8759 bytes)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Starting task 4.0 in stage 7.0 (TID 53, localhost, executor driver, partition 4, PROCESS_LOCAL, 8759 bytes)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Starting task 5.0 in stage 7.0 (TID 54, localhost, executor driver, partition 5, PROCESS_LOCAL, 8759 bytes)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Starting task 6.0 in stage 7.0 (TID 55, localhost, executor driver, partition 6, PROCESS_LOCAL, 8759 bytes)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Starting task 7.0 in stage 7.0 (TID 56, localhost, executor driver, partition 7, PROCESS_LOCAL, 8759 bytes)
2024-04-12 20:05:34 INFO Executor:54 - Running task 1.0 in stage 7.0 (TID 50)
2024-04-12 20:05:34 INFO Executor:54 - Running task 0.0 in stage 7.0 (TID 49)
2024-04-12 20:05:34 INFO Executor:54 - Running task 3.0 in stage 7.0 (TID 52)
2024-04-12 20:05:34 INFO Executor:54 - Running task 4.0 in stage 7.0 (TID 53)
2024-04-12 20:05:34 INFO Executor:54 - Running task 6.0 in stage 7.0 (TID 55)
2024-04-12 20:05:34 INFO Executor:54 - Running task 7.0 in stage 7.0 (TID 56)
2024-04-12 20:05:34 INFO Executor:54 - Running task 5.0 in stage 7.0 (TID 54)
2024-04-12 20:05:34 INFO Executor:54 - Running task 2.0 in stage 7.0 (TID 51)
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_13_2 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_13_1 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_13_0 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_13_4 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_13_5 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_14_4 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_14_1 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_13_3 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_14_0 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_14_2 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_24_0 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_14_3 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_24_1 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_14_5 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_24_5 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_24_4 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_24_3 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_24_2 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_13_7 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_14_7 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_24_7 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_13_6 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_14_6 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_24_6 locally
2024-04-12 20:05:34 INFO Executor:54 - Finished task 4.0 in stage 7.0 (TID 53). 1426 bytes result sent to driver
2024-04-12 20:05:34 INFO Executor:54 - Finished task 0.0 in stage 7.0 (TID 49). 2284 bytes result sent to driver
2024-04-12 20:05:34 INFO Executor:54 - Finished task 1.0 in stage 7.0 (TID 50). 1734 bytes result sent to driver
2024-04-12 20:05:34 INFO TaskSetManager:54 - Finished task 4.0 in stage 7.0 (TID 53) in 94 ms on localhost (executor driver) (1/8)
2024-04-12 20:05:34 INFO Executor:54 - Finished task 3.0 in stage 7.0 (TID 52). 1820 bytes result sent to driver
2024-04-12 20:05:34 INFO Executor:54 - Finished task 5.0 in stage 7.0 (TID 54). 1844 bytes result sent to driver
2024-04-12 20:05:34 INFO TaskSetManager:54 - Finished task 0.0 in stage 7.0 (TID 49) in 112 ms on localhost (executor driver) (2/8)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Finished task 1.0 in stage 7.0 (TID 50) in 113 ms on localhost (executor driver) (3/8)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Finished task 3.0 in stage 7.0 (TID 52) in 115 ms on localhost (executor driver) (4/8)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Finished task 5.0 in stage 7.0 (TID 54) in 115 ms on localhost (executor driver) (5/8)
2024-04-12 20:05:34 INFO Executor:54 - Finished task 2.0 in stage 7.0 (TID 51). 2217 bytes result sent to driver
2024-04-12 20:05:34 INFO Executor:54 - Finished task 7.0 in stage 7.0 (TID 56). 1997 bytes result sent to driver
2024-04-12 20:05:34 INFO Executor:54 - Finished task 6.0 in stage 7.0 (TID 55). 2107 bytes result sent to driver
2024-04-12 20:05:34 INFO TaskSetManager:54 - Finished task 7.0 in stage 7.0 (TID 56) in 120 ms on localhost (executor driver) (6/8)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Finished task 2.0 in stage 7.0 (TID 51) in 122 ms on localhost (executor driver) (7/8)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Finished task 6.0 in stage 7.0 (TID 55) in 120 ms on localhost (executor driver) (8/8)
2024-04-12 20:05:34 INFO TaskSchedulerImpl:54 - Removed TaskSet 7.0, whose tasks have all completed, from pool
2024-04-12 20:05:34 INFO DAGScheduler:54 - ResultStage 7 (collect at KMeans.scala:382) finished in 0.138 s
2024-04-12 20:05:34 INFO DAGScheduler:54 - Job 7 finished: collect at KMeans.scala:382, took 0.154571 s
2024-04-12 20:05:34 INFO MapPartitionsRDD:54 - Removing RDD 24 from persistence list
2024-04-12 20:05:34 INFO BlockManager:54 - Removing RDD 24
2024-04-12 20:05:34 INFO TorrentBroadcast:54 - Destroying Broadcast(7) (from destroy at KMeans.scala:389)
2024-04-12 20:05:34 INFO TorrentBroadcast:54 - Destroying Broadcast(10) (from destroy at KMeans.scala:389)
2024-04-12 20:05:34 INFO BlockManagerInfo:54 - Removed broadcast_7_piece0 on 10.0.0.154:53058 in memory (size: 412.0 B, free: 102.5 MB)
2024-04-12 20:05:34 INFO BlockManagerInfo:54 - Removed broadcast_10_piece0 on 10.0.0.154:53058 in memory (size: 2.1 KB, free: 102.5 MB)
2024-04-12 20:05:34 INFO MemoryStore:54 - Block broadcast_13 stored as values in memory (estimated size 5.9 KB, free 102.1 MB)
2024-04-12 20:05:34 INFO MemoryStore:54 - Block broadcast_13_piece0 stored as bytes in memory (estimated size 4.0 KB, free 102.1 MB)
2024-04-12 20:05:34 INFO BlockManagerInfo:54 - Added broadcast_13_piece0 in memory on 10.0.0.154:53058 (size: 4.0 KB, free: 102.5 MB)
2024-04-12 20:05:34 INFO SparkContext:54 - Created broadcast 13 from broadcast at KMeans.scala:399
2024-04-12 20:05:34 INFO SparkContext:54 - Starting job: countByValue at KMeans.scala:400
2024-04-12 20:05:34 INFO DAGScheduler:54 - Registering RDD 29 (countByValue at KMeans.scala:400)
2024-04-12 20:05:34 INFO DAGScheduler:54 - Got job 8 (countByValue at KMeans.scala:400) with 8 output partitions
2024-04-12 20:05:34 INFO DAGScheduler:54 - Final stage: ResultStage 9 (countByValue at KMeans.scala:400)
2024-04-12 20:05:34 INFO DAGScheduler:54 - Parents of final stage: List(ShuffleMapStage 8)
2024-04-12 20:05:34 INFO DAGScheduler:54 - Missing parents: List(ShuffleMapStage 8)
2024-04-12 20:05:34 INFO DAGScheduler:54 - Submitting ShuffleMapStage 8 (MapPartitionsRDD[29] at countByValue at KMeans.scala:400), which has no missing parents
2024-04-12 20:05:34 INFO MemoryStore:54 - Block broadcast_14 stored as values in memory (estimated size 32.4 KB, free 102.1 MB)
2024-04-12 20:05:34 INFO MemoryStore:54 - Block broadcast_14_piece0 stored as bytes in memory (estimated size 14.7 KB, free 102.1 MB)
2024-04-12 20:05:34 INFO BlockManagerInfo:54 - Added broadcast_14_piece0 in memory on 10.0.0.154:53058 (size: 14.7 KB, free: 102.4 MB)
2024-04-12 20:05:34 INFO SparkContext:54 - Created broadcast 14 from broadcast at DAGScheduler.scala:1039
2024-04-12 20:05:34 INFO DAGScheduler:54 - Submitting 8 missing tasks from ShuffleMapStage 8 (MapPartitionsRDD[29] at countByValue at KMeans.scala:400) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7))
2024-04-12 20:05:34 INFO TaskSchedulerImpl:54 - Adding task set 8.0 with 8 tasks
2024-04-12 20:05:34 INFO TaskSetManager:54 - Starting task 0.0 in stage 8.0 (TID 57, localhost, executor driver, partition 0, PROCESS_LOCAL, 8652 bytes)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Starting task 1.0 in stage 8.0 (TID 58, localhost, executor driver, partition 1, PROCESS_LOCAL, 8652 bytes)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Starting task 2.0 in stage 8.0 (TID 59, localhost, executor driver, partition 2, PROCESS_LOCAL, 8652 bytes)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Starting task 3.0 in stage 8.0 (TID 60, localhost, executor driver, partition 3, PROCESS_LOCAL, 8652 bytes)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Starting task 4.0 in stage 8.0 (TID 61, localhost, executor driver, partition 4, PROCESS_LOCAL, 8652 bytes)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Starting task 5.0 in stage 8.0 (TID 62, localhost, executor driver, partition 5, PROCESS_LOCAL, 8652 bytes)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Starting task 6.0 in stage 8.0 (TID 63, localhost, executor driver, partition 6, PROCESS_LOCAL, 8652 bytes)
2024-04-12 20:05:34 INFO TaskSetManager:54 - Starting task 7.0 in stage 8.0 (TID 64, localhost, executor driver, partition 7, PROCESS_LOCAL, 8652 bytes)
2024-04-12 20:05:34 INFO Executor:54 - Running task 5.0 in stage 8.0 (TID 62)
2024-04-12 20:05:34 INFO Executor:54 - Running task 3.0 in stage 8.0 (TID 60)
2024-04-12 20:05:34 INFO Executor:54 - Running task 1.0 in stage 8.0 (TID 58)
2024-04-12 20:05:34 INFO Executor:54 - Running task 2.0 in stage 8.0 (TID 59)
2024-04-12 20:05:34 INFO Executor:54 - Running task 0.0 in stage 8.0 (TID 57)
2024-04-12 20:05:34 INFO Executor:54 - Running task 7.0 in stage 8.0 (TID 64)
2024-04-12 20:05:34 INFO Executor:54 - Running task 4.0 in stage 8.0 (TID 61)
2024-04-12 20:05:34 INFO Executor:54 - Running task 6.0 in stage 8.0 (TID 63)
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_13_4 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_13_3 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_14_3 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_13_7 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_13_2 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_13_1 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_14_7 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_13_6 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_13_0 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_13_5 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_14_0 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_14_6 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_14_1 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_14_2 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_14_4 locally
2024-04-12 20:05:34 INFO BlockManager:54 - Found block rdd_14_5 locally
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 193
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 206
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 194
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 213
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 215
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 210
2024-04-12 20:05:35 INFO BlockManagerInfo:54 - Removed broadcast_12_piece0 on 10.0.0.154:53058 in memory (size: 14.4 KB, free: 102.5 MB)
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 198
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 207
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 201
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 196
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 199
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 200
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 192
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 205
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 211
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 191
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 212
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 195
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 197
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 209
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 202
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 203
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 204
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 214
2024-04-12 20:05:35 INFO ContextCleaner:54 - Cleaned accumulator 208
2024-04-12 20:05:35 INFO Executor:54 - Finished task 6.0 in stage 8.0 (TID 63). 1801 bytes result sent to driver
2024-04-12 20:05:35 INFO Executor:54 - Finished task 4.0 in stage 8.0 (TID 61). 1801 bytes result sent to driver
2024-04-12 20:05:35 INFO Executor:54 - Finished task 0.0 in stage 8.0 (TID 57). 1801 bytes result sent to driver
2024-04-12 20:05:35 INFO Executor:54 - Finished task 3.0 in stage 8.0 (TID 60). 1801 bytes result sent to driver
2024-04-12 20:05:35 INFO Executor:54 - Finished task 2.0 in stage 8.0 (TID 59). 1801 bytes result sent to driver
2024-04-12 20:05:35 INFO Executor:54 - Finished task 5.0 in stage 8.0 (TID 62). 1801 bytes result sent to driver
2024-04-12 20:05:35 INFO Executor:54 - Finished task 1.0 in stage 8.0 (TID 58). 1801 bytes result sent to driver
2024-04-12 20:05:35 INFO Executor:54 - Finished task 7.0 in stage 8.0 (TID 64). 1801 bytes result sent to driver
2024-04-12 20:05:35 INFO TaskSetManager:54 - Finished task 1.0 in stage 8.0 (TID 58) in 1114 ms on localhost (executor driver) (1/8)
2024-04-12 20:05:35 INFO TaskSetManager:54 - Finished task 5.0 in stage 8.0 (TID 62) in 1114 ms on localhost (executor driver) (2/8)
2024-04-12 20:05:35 INFO TaskSetManager:54 - Finished task 0.0 in stage 8.0 (TID 57) in 1127 ms on localhost (executor driver) (3/8)
2024-04-12 20:05:35 INFO TaskSetManager:54 - Finished task 2.0 in stage 8.0 (TID 59) in 1116 ms on localhost (executor driver) (4/8)
2024-04-12 20:05:35 INFO TaskSetManager:54 - Finished task 6.0 in stage 8.0 (TID 63) in 1125 ms on localhost (executor driver) (5/8)
2024-04-12 20:05:35 INFO TaskSetManager:54 - Finished task 7.0 in stage 8.0 (TID 64) in 1126 ms on localhost (executor driver) (6/8)
2024-04-12 20:05:35 INFO TaskSetManager:54 - Finished task 3.0 in stage 8.0 (TID 60) in 1126 ms on localhost (executor driver) (7/8)
2024-04-12 20:05:35 INFO TaskSetManager:54 - Finished task 4.0 in stage 8.0 (TID 61) in 1126 ms on localhost (executor driver) (8/8)
2024-04-12 20:05:35 INFO TaskSchedulerImpl:54 - Removed TaskSet 8.0, whose tasks have all completed, from pool
2024-04-12 20:05:35 INFO DAGScheduler:54 - ShuffleMapStage 8 (countByValue at KMeans.scala:400) finished in 1.155 s
2024-04-12 20:05:35 INFO DAGScheduler:54 - looking for newly runnable stages
2024-04-12 20:05:35 INFO DAGScheduler:54 - running: Set()
2024-04-12 20:05:35 INFO DAGScheduler:54 - waiting: Set(ResultStage 9)
2024-04-12 20:05:35 INFO DAGScheduler:54 - failed: Set()
2024-04-12 20:05:35 INFO DAGScheduler:54 - Submitting ResultStage 9 (ShuffledRDD[30] at countByValue at KMeans.scala:400), which has no missing parents
2024-04-12 20:05:35 INFO MemoryStore:54 - Block broadcast_15 stored as values in memory (estimated size 3.2 KB, free 102.1 MB)
2024-04-12 20:05:35 INFO MemoryStore:54 - Block broadcast_15_piece0 stored as bytes in memory (estimated size 1983.0 B, free 102.1 MB)
2024-04-12 20:05:35 INFO BlockManagerInfo:54 - Added broadcast_15_piece0 in memory on 10.0.0.154:53058 (size: 1983.0 B, free: 102.5 MB)
2024-04-12 20:05:35 INFO SparkContext:54 - Created broadcast 15 from broadcast at DAGScheduler.scala:1039
2024-04-12 20:05:35 INFO DAGScheduler:54 - Submitting 8 missing tasks from ResultStage 9 (ShuffledRDD[30] at countByValue at KMeans.scala:400) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7))
2024-04-12 20:05:35 INFO TaskSchedulerImpl:54 - Adding task set 9.0 with 8 tasks
2024-04-12 20:05:35 INFO TaskSetManager:54 - Starting task 0.0 in stage 9.0 (TID 65, localhost, executor driver, partition 0, ANY, 7649 bytes)
2024-04-12 20:05:35 INFO TaskSetManager:54 - Starting task 1.0 in stage 9.0 (TID 66, localhost, executor driver, partition 1, ANY, 7649 bytes)
2024-04-12 20:05:35 INFO TaskSetManager:54 - Starting task 2.0 in stage 9.0 (TID 67, localhost, executor driver, partition 2, ANY, 7649 bytes)
2024-04-12 20:05:35 INFO TaskSetManager:54 - Starting task 3.0 in stage 9.0 (TID 68, localhost, executor driver, partition 3, ANY, 7649 bytes)
2024-04-12 20:05:35 INFO TaskSetManager:54 - Starting task 4.0 in stage 9.0 (TID 69, localhost, executor driver, partition 4, ANY, 7649 bytes)
2024-04-12 20:05:35 INFO TaskSetManager:54 - Starting task 5.0 in stage 9.0 (TID 70, localhost, executor driver, partition 5, ANY, 7649 bytes)
2024-04-12 20:05:35 INFO TaskSetManager:54 - Starting task 6.0 in stage 9.0 (TID 71, localhost, executor driver, partition 6, ANY, 7649 bytes)
2024-04-12 20:05:35 INFO TaskSetManager:54 - Starting task 7.0 in stage 9.0 (TID 72, localhost, executor driver, partition 7, ANY, 7649 bytes)
2024-04-12 20:05:35 INFO Executor:54 - Running task 1.0 in stage 9.0 (TID 66)
2024-04-12 20:05:35 INFO Executor:54 - Running task 3.0 in stage 9.0 (TID 68)
2024-04-12 20:05:35 INFO Executor:54 - Running task 2.0 in stage 9.0 (TID 67)
2024-04-12 20:05:35 INFO Executor:54 - Running task 0.0 in stage 9.0 (TID 65)
2024-04-12 20:05:35 INFO Executor:54 - Running task 7.0 in stage 9.0 (TID 72)
2024-04-12 20:05:35 INFO Executor:54 - Running task 4.0 in stage 9.0 (TID 69)
2024-04-12 20:05:35 INFO Executor:54 - Running task 6.0 in stage 9.0 (TID 71)
2024-04-12 20:05:35 INFO Executor:54 - Running task 5.0 in stage 9.0 (TID 70)
2024-04-12 20:05:35 INFO ShuffleBlockFetcherIterator:54 - Getting 8 non-empty blocks out of 8 blocks
2024-04-12 20:05:35 INFO ShuffleBlockFetcherIterator:54 - Getting 8 non-empty blocks out of 8 blocks
2024-04-12 20:05:35 INFO ShuffleBlockFetcherIterator:54 - Getting 8 non-empty blocks out of 8 blocks
2024-04-12 20:05:35 INFO ShuffleBlockFetcherIterator:54 - Getting 8 non-empty blocks out of 8 blocks
2024-04-12 20:05:35 INFO ShuffleBlockFetcherIterator:54 - Getting 8 non-empty blocks out of 8 blocks
2024-04-12 20:05:35 INFO ShuffleBlockFetcherIterator:54 - Getting 8 non-empty blocks out of 8 blocks
2024-04-12 20:05:35 INFO ShuffleBlockFetcherIterator:54 - Getting 8 non-empty blocks out of 8 blocks
2024-04-12 20:05:35 INFO ShuffleBlockFetcherIterator:54 - Getting 8 non-empty blocks out of 8 blocks
2024-04-12 20:05:35 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 8 ms
2024-04-12 20:05:35 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 8 ms
2024-04-12 20:05:35 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 8 ms
2024-04-12 20:05:35 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 8 ms
2024-04-12 20:05:35 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 8 ms
2024-04-12 20:05:35 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 8 ms
2024-04-12 20:05:35 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 8 ms
2024-04-12 20:05:35 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 8 ms
2024-04-12 20:05:35 INFO Executor:54 - Finished task 2.0 in stage 9.0 (TID 67). 1523 bytes result sent to driver
2024-04-12 20:05:35 INFO Executor:54 - Finished task 1.0 in stage 9.0 (TID 66). 1523 bytes result sent to driver
2024-04-12 20:05:35 INFO Executor:54 - Finished task 5.0 in stage 9.0 (TID 70). 1493 bytes result sent to driver
2024-04-12 20:05:35 INFO Executor:54 - Finished task 0.0 in stage 9.0 (TID 65). 1523 bytes result sent to driver
2024-04-12 20:05:35 INFO Executor:54 - Finished task 3.0 in stage 9.0 (TID 68). 1493 bytes result sent to driver
2024-04-12 20:05:35 INFO Executor:54 - Finished task 7.0 in stage 9.0 (TID 72). 1493 bytes result sent to driver
2024-04-12 20:05:35 INFO Executor:54 - Finished task 6.0 in stage 9.0 (TID 71). 1493 bytes result sent to driver
2024-04-12 20:05:35 INFO Executor:54 - Finished task 4.0 in stage 9.0 (TID 69). 1493 bytes result sent to driver
2024-04-12 20:05:35 INFO TaskSetManager:54 - Finished task 2.0 in stage 9.0 (TID 67) in 262 ms on localhost (executor driver) (1/8)