Data Challenge Overview
Data challenge is a new addition to the CODS-COMAD conference in the form of a pre-conference competition. The challenge aims to attract teams from the Data Science, Data mining and Machine Learning community to participate. We expect students, practitioners, and researchers to form teams and participate in the competition.
Discover the sequence of academic videos through our data challenge, aimed at addressing the navigation of online learning content. As educational material grows, students often struggle to find the optimal learning path.
Participate in the data challenge by completing this registration form. Organizers will reach out to you with a Kaggle link after the registration deadline.
- First place (team prize): ₹ 50,000/-
- Second place (team prize): ₹ 25,000/-
(Second place prize will be awarded only if the number of participating teams are 25 or more)
In addition to the cash prize, winners and runners-up will be presented certificates during the conference award ceremony. They will also get time in the conference program to present their solution(s). One team member from the winning team will receive a complimentary registration to the conference. Student member(s) from the winning team will also be eligible for the CODS-COMAD student travel grant.
What's the Challenge?
- Dataset: VID-REQ, as detailed in "Auto-req: Automatic detection of pre-requisite dependencies between academic videos." Learn more at ACL proceedings link.
- Collaboration: A combined endeavor by Extramarks Education's data team and IIIT Delhi.
- Focus: Tailored to Indian K12 science and math, with content from Extramarks Education.
Why is this Important?
Unraveling pre-requisites in academic videos unites machine learning, deep learning, and NLP. This groundbreaking research emphasizes our dedication to democratizing education through AI, with a focus on the unique challenges of the Indian K12 context. Join us in solving this engaging problem, fostering innovation, and developing diverse, accessible solutions.
- Availability & Confidentiality:
- The VID-REQ pre-requisite dataset is open for public access at DATA Link with README.
- Test set of 500 hand-labeled edges remains confidential.
- Dataset Composition:
- Original set contains 2,797 labeled pre-requisite edges.
- An additional 500 edges, labeled in-house, are included for this challenge.
- Evaluation Metric:
- Solutions will be gauged using the weighted F1 score, suitable for binary classification tasks with minor class imbalances.
- Licensing & Permissions:
- Dataset usage complies with Creative Commons Attribution-NonCommercial-Share Alike 3.0 International License per ACL guidelines.
- Dataset Explanation:
- Total of 2,797 hand-labeled edges: 1,684 non-prerequisite (label: 0) and 1,113 prerequisite (label: 1).
- Labels were assigned by experts after reviewing video pairs for prerequisite relationships.
- Each video comes with a distinct taxonomy, sourced from K12 textbooks via a PDF parser.
- The Inter-annotator agreement, as per Cohen’s Kappa coefficient, stands strong at 0.64.
- Dataset Contents:
- Includes annotated video edges, titles, taxonomies, transcripts, binary labels, and our extracted features.
- User Opportunity:
- This dataset serves as a foundation for modeling prerequisite video relations. We offer our extracted features for beginners and raw data (like video transcripts) for advanced users aiming to refine prediction accuracy.
Tasks & Problem Overview
Dive into a challenge that tests your creativity, analytical prowess, and problem-solving skills. We're eager to see your innovative approaches in identifying the correct pre-requisite edges among potential edges.
Is watching Video A crucial before diving into Video B? In essence, does Video A lay the foundational knowledge that enhances comprehension of Video B?
- Utilize the provided features and extract more from the public video transcripts if needed.
- Train a machine learning model to predict one of the two outcomes for each dataset edge:
- 0 - Not a prerequisite.
- 1 - Indeed a prerequisite.
Resources: We'll provide the essential training data with labels and the label-excluded testing data. Once you've trained your model on our dataset, submit your predictions for the testing data on the designated platform for evaluation.
- Platform: Hosted on Kaggle, covering dataset release, leaderboard, registration, submission, and evaluation.
Compute Platform Guidelines
- Flexible Environment: Participants have the freedom to choose their computing setup.
- Cloud Platforms: Kaggle and Google's Colab are available for participants, offering in-browser work without the need for advanced hardware.
- Personal Resources & Advanced Compute: Using personal setups or additional computational power is permitted. As a reference, prior training utilized:
- CPU: Intel i9-12900K
- GPU: Nvidia RTX 3060 12 GB
- Note: Higher-tier systems may expedite processing, but solution quality depends on creativity and model selection. Large models like LLMs or BERT may need high-performance GPUs. For setup assistance, contact the organizing team.
Evaluation and Leaderboard
- Metric: Submissions evaluated using Weighted F1 score on a 500-edge test set, ensuring it mirrors the training set's distribution.
- Real-World Impact: A live demo during the conference will showcase the algorithms' ability to identify prerequisite video relationships.
- Evaluation Objective: Blend of online and real-world assessments to ensure practical and theoretical excellence.
Timeline and Implementation Details
|1.|| Registration Deadline:|| August 31, 2023 (AoE Time)|
|2.||Competition Start:|| September 1, 2023. Public release of training data (2,797 edges), competition guidelines, evaluation metrics, and submission format.|
|3.|| Test Data Release:|| October 1, 2023. 500-edge test set release; predictions required without public labels.|
|4.|| Submission Deadline:|| November 30, 2023.|
|5.|| Winner Notification:|| December 10, 2023.|
|6.|| Official Announcement:|| December 12, 2023.|
|7.|| Presentation & Demo:|| January 6, 2024.|
Competition Rules & Regulations
Platform & Submissions
- Competition hosted on Kaggle with a public leaderboard.
- Maximum of 15 submissions per team, daily.
Evaluation & Dataset
- Teams may have up to 3 members.
- Composition is fixed once evaluation starts.
Papers & Results
- Weighted F1 metric for the public leaderboard.
- Redistribution of dataset is disallowed without citation.
Liability & Integrity
- Top teams invited to submit system description papers.
- Data preprocessing allowed but must be documented.
- Public release of scores is mandatory; reporting in papers encouraged.
- No guarantees on dataset accuracy by organizers.
- Incomplete or deceptive submissions can lead to withheld scores.
Open-Source & Public Tools Policy
- Open-source software and public APIs are encouraged.
- Abide by terms and service limits of these tools.
- Cite and acknowledge all used resources.
- Ensure code clarity and reproducibility.
- Avoid proprietary/confidential tools without permission.
Pre-Trained Model Policy
- Public models are allowed with citation.
- Proprietary models disallowed without owner consent.
- Always disclose pre-trained model usage and modifications.
- Models should be accessible and reproducible.
- If a model is trained on a similar dataset, state and justify.
LLM Usage Policy
- LLMs like ChatGPT can be used for coding help.
- Encourage personal creativity in model and feature selection.
- Mention LLM usage in submission.
- Properly attribute any ChatGPT-assisted code.
Code Sharing Requirement
- Submit final solution source code.
- You retain ownership of your code.
- Include scripts and instructions for execution.
- Code should be well-documented with a clear README.
- The aim is for others to reproduce your solution.
Dispute & Arbitration Terms
- Disqualification: Organizers can disqualify participants for misconduct.
- Plagiarism: Immediate disqualification for plagiarism.
- Novelty: Original solutions encouraged; build upon existing ones with acknowledgment.
- Dispute Resolution: Organizer decisions are final.
- Arbitration: For unresolved disputes, organizers choose the location and arbitrator.
- Governing Law: Host country laws apply to competition terms.
By participating, you fully accept these terms.
Register here. Organizers will reach out to you with a Kaggle link after the registration deadline.