=================================================================================== TAC 2020 STREAMING MULTIMEDIA KNOWLEDGE BASE POPULATION EVALUATION RESULTS (TASK 1) =================================================================================== Team ID: OPERA Organization(s): Carnegie Mellon University; USC Information Sciences Institute (ISI) ******************** Run: teamcmu-ev-202010161428 English Evaluation Results: Score Metric 0.3633 CoreferenceMetric_ENG_Entities_F1 0.3328 TypeMetricV1_ENG_Entities_F1 0.3051 CoreferenceMetric_ENG_Events_F1 0.3018 TypeMetricV1_ENG_Events_F1 0.2552 ArgumentMetricV1_ENG_Events_F1 0.1061 FrameMetric_ENG_Events_F1 0.0805 FrameMetric_ENG_Relations_F1 0.1098 TemporalMetric_ENG_Events_S 0.0028 TemporalMetric_ENG_Relations_S 0.0968 FrameMetric_ENG_F1 Spanish Evaluation Results: Score Metric 0.2820 CoreferenceMetric_SPA_Entities_F1 0.2653 TypeMetricV1_SPA_Entities_F1 0.1755 CoreferenceMetric_SPA_Events_F1 0.1469 TypeMetricV1_SPA_Events_F1 0.1913 ArgumentMetricV1_SPA_Events_F1 0.0489 FrameMetric_SPA_Events_F1 0.0252 FrameMetric_SPA_Relations_F1 0.0376 TemporalMetric_SPA_Events_S 0.0000 TemporalMetric_SPA_Relations_S 0.0407 FrameMetric_SPA_F1 Russian Evaluation Results: Score Metric 0.3005 CoreferenceMetric_RUS_Entities_F1 0.2351 TypeMetricV1_RUS_Entities_F1 0.2333 CoreferenceMetric_RUS_Events_F1 0.1948 TypeMetricV1_RUS_Events_F1 0.1655 ArgumentMetricV1_RUS_Events_F1 0.0715 FrameMetric_RUS_Events_F1 0.0000 FrameMetric_RUS_Relations_F1 0.0574 TemporalMetric_RUS_Events_S 0.0000 TemporalMetric_RUS_Relations_S 0.0487 FrameMetric_RUS_F1 Tri-lingual (English, Spanish, Russian) Evaluation Results: Score Metric 0.3126 CoreferenceMetric_Entities_F1 0.2826 TypeMetricV1_Entities_F1 0.2328 CoreferenceMetric_Events_F1 0.2276 TypeMetricV1_Events_F1 0.2035 ArgumentMetricV1_Events_F1 0.0795 FrameMetric_Events_F1 0.0462 FrameMetric_Relations_F1 0.0756 TemporalMetric_Events_S 0.0017 TemporalMetric_Relations_S 0.0680 FrameMetric_F1 ******************** Run: teamcmu-ta1-202009301423 English Evaluation Results: Score Metric 0.3655 CoreferenceMetric_ENG_Entities_F1 0.3416 TypeMetricV1_ENG_Entities_F1 0.3006 CoreferenceMetric_ENG_Events_F1 0.2984 TypeMetricV1_ENG_Events_F1 0.2545 ArgumentMetricV1_ENG_Events_F1 0.0992 FrameMetric_ENG_Events_F1 0.0000 FrameMetric_ENG_Relations_F1 0.1133 TemporalMetric_ENG_Events_S 0.0000 TemporalMetric_ENG_Relations_S 0.0679 FrameMetric_ENG_F1 Spanish Evaluation Results: Score Metric 0.2770 CoreferenceMetric_SPA_Entities_F1 0.2640 TypeMetricV1_SPA_Entities_F1 0.1910 CoreferenceMetric_SPA_Events_F1 0.1918 TypeMetricV1_SPA_Events_F1 0.1457 ArgumentMetricV1_SPA_Events_F1 0.0418 FrameMetric_SPA_Events_F1 0.0000 FrameMetric_SPA_Relations_F1 0.0232 TemporalMetric_SPA_Events_S 0.0000 TemporalMetric_SPA_Relations_S 0.0296 FrameMetric_SPA_F1 Russian Evaluation Results: Score Metric 0.2062 CoreferenceMetric_RUS_Entities_F1 0.2092 TypeMetricV1_RUS_Entities_F1 0.2262 CoreferenceMetric_RUS_Events_F1 0.1903 TypeMetricV1_RUS_Events_F1 0.1545 ArgumentMetricV1_RUS_Events_F1 0.0639 FrameMetric_RUS_Events_F1 0.0000 FrameMetric_RUS_Relations_F1 0.0334 TemporalMetric_RUS_Events_S 0.0000 TemporalMetric_RUS_Relations_S 0.0454 FrameMetric_RUS_F1 Tri-lingual (English, Spanish, Russian) Evaluation Results: Score Metric 0.2835 CoreferenceMetric_Entities_F1 0.2751 TypeMetricV1_Entities_F1 0.2354 CoreferenceMetric_Events_F1 0.2367 TypeMetricV1_Events_F1 0.1820 ArgumentMetricV1_Events_F1 0.0709 FrameMetric_Events_F1 0.0000 FrameMetric_Relations_F1 0.0682 TemporalMetric_Events_S 0.0000 TemporalMetric_Relations_S 0.0495 FrameMetric_F1 ******************** Run: teamcmu-ta1-202010061159 English Evaluation Results: Score Metric 0.3618 CoreferenceMetric_ENG_Entities_F1 0.3300 TypeMetricV1_ENG_Entities_F1 0.3001 CoreferenceMetric_ENG_Events_F1 0.2938 TypeMetricV1_ENG_Events_F1 0.2682 ArgumentMetricV1_ENG_Events_F1 0.1025 FrameMetric_ENG_Events_F1 0.0828 FrameMetric_ENG_Relations_F1 0.1057 TemporalMetric_ENG_Events_S 0.0000 TemporalMetric_ENG_Relations_S 0.0958 FrameMetric_ENG_F1 Spanish Evaluation Results: Score Metric 0.2791 CoreferenceMetric_SPA_Entities_F1 0.2624 TypeMetricV1_SPA_Entities_F1 0.1834 CoreferenceMetric_SPA_Events_F1 0.1643 TypeMetricV1_SPA_Events_F1 0.1408 ArgumentMetricV1_SPA_Events_F1 0.0387 FrameMetric_SPA_Events_F1 0.0254 FrameMetric_SPA_Relations_F1 0.0344 TemporalMetric_SPA_Events_S 0.0000 TemporalMetric_SPA_Relations_S 0.0341 FrameMetric_SPA_F1 Russian Evaluation Results: Score Metric 0.2734 CoreferenceMetric_RUS_Entities_F1 0.2362 TypeMetricV1_RUS_Entities_F1 0.2322 CoreferenceMetric_RUS_Events_F1 0.1924 TypeMetricV1_RUS_Events_F1 0.1693 ArgumentMetricV1_RUS_Events_F1 0.0618 FrameMetric_RUS_Events_F1 0.0000 FrameMetric_RUS_Relations_F1 0.0334 TemporalMetric_RUS_Events_S 0.0000 TemporalMetric_RUS_Relations_S 0.0422 FrameMetric_RUS_F1 Tri-lingual (English, Spanish, Russian) Evaluation Results: Score Metric 0.3030 CoreferenceMetric_Entities_F1 0.2815 TypeMetricV1_Entities_F1 0.2340 CoreferenceMetric_Events_F1 0.2310 TypeMetricV1_Events_F1 0.1887 ArgumentMetricV1_Events_F1 0.0733 FrameMetric_Events_F1 0.0470 FrameMetric_Relations_F1 0.0685 TemporalMetric_Events_S 0.0000 TemporalMetric_Relations_S 0.0645 FrameMetric_F1