Products

Research

Enterprise

For Developers

Company

Playground

Talk to Sales

Tutorial

Tutorial

Tutorial

Building An Olympic Video Classification Application with Twelve Labs

Hrishikesh Yadav

Hrishikesh Yadav

Hrishikesh Yadav

The Olympics Video Clips Classification Application is a powerful tool designed to categorize various Olympic sports using video clips.

The Olympics Video Clips Classification Application is a powerful tool designed to categorize various Olympic sports using video clips.

Join our newsletter

Receive the latest advancements, tutorials, and industry insights in video understanding

Sep 3, 2024

Sep 3, 2024

Sep 3, 2024

15 Min

15 Min

15 Min

Copy link to article

Copy link to article

Copy link to article

Introduction

Ever dreamed of winning a medal for organizing Olympic video footage? 🥇 Well, now's your chance to stand on the podium!

The Olympics Video Clips Classification Application aims to streamline the tedious process of categorizing sports footage. Using Twelve Labs' Marengo 2.6 Embedding Model, this app swiftly classifies Olympic sports from video clips.

By analyzing on-screen text, conversations, and visuals, the application categorizes video clips with ease. This tutorial will guide you through creating a Streamlit application that revolutionizes how scholars, sports enthusiasts, and broadcasters interact with Olympic content. You'll learn to build an app that classifies videos based on user-defined categories. A detailed demonstration video of the application is provided below.

You can explore the demo of the application here: Olympics Application. You can also play with it via this Replit template.

‍

Prerequisite

  • Generate an API key by signing up at the Twelve Labs Playground.

  • Find the repository for the notebooks and this application on GitHub.

  • The Streamlit application uses Python, HTML, and JavaScript.


Working of the Application

This section outlines the application flow for classifying Olympic video clips.

The application uses a classification search engine, which has various potential uses. In this task, we'll focus on classifying and retrieving Olympic sports video clips. The first crucial step is creating an index.

  • To do this, visit the Twelve Labs Playground.

  • Create a New Index and select Marengo 2.6 (Embedding Engine), which is suitable for indexing and classification.

  • Once all sports videos are uploaded to this index, we can proceed.

The following section details how to create and implement an Index ID step by step. Here's an overview of the general application workflow:

  • Users have two options: they can either select the type of sport they want to view from a multi-select menu or add their own custom classes.

  • The selected class is then sent to the classify endpoint, along with the INDEX ID, to retrieve and classify all relevant video clips. Additional parameters, such as include_clips, are also sent to this endpoint.

  • The response includes the Video ID, class, confidence level, start and end times, thumbnail URL, and more.

  • To obtain the video URL associated with the resulting Video ID, we need to access the video info endpoint.

Preparation Steps

  • Sign up and create an Index on the Twelve Labs Playground.

  • Choose the appropriate option for classification: Marengo 2.6 (Embedding Engine). This engine excels in video search and classification, providing a robust foundation for video understanding.

  • Upload sports video clips. If needed, use the sample clips provided for this tutorial: Sports Clips.

With these steps completed, you're now ready to dive in and develop the application!

Twelvelabs_API=your_api_key_here
API_URL=your_api_url_here
INDEX_ID=your_index_id_here

Walkthrough for Classification

1 - Importing, Validating, and Setting Up the Client

We begin by importing the necessary libraries and setting up environment variables to create the correct environment. To get started:

import os
import requests
import glob
import requests
from twelvelabs import TwelveLabs
from twelvelabs.models.task import Task
import dotenv


API_KEY=os.getenv("Twelvelabs_API")
API_URL=os.getenv("API_URL")
INDEX_ID=os.getenv("INDEX_ID")

client = TwelveLabs(api_key=API_KEY)

# URL of the /indexes endpoint
INDEXES_URL = f"{API_URL}/indexes"
     
# Setting headers variables

default_header = {
    "x-api-key": API_KEY
}

The dependencies have been imported, environment variables loaded correctly, and the API client initialized with the proper credentials. The base URL for accessing indexes and default headers for API requests have also been set up.

With this foundation in place, you're now ready to proceed with more specific classification tasks, such as creating indexes.

‍

2 - Setting up the Classes

Below is the list of classes for classifying Olympic sports clips:

# Categories for the classification of the Olympics Sports

CLASSES = [
    {
        "name": "AquaticSports",
        "prompts": [
            "swimming competition",
            "diving event",
            "water polo match",
            "synchronized swimming",
            "open water swimming"
        ]
    },
    {
        "name": "AthleticEvents",
        "prompts": [
            "track and field",
            "marathon running",
            "long jump competition",
            "javelin throw",
            "high jump event"
        ]
    },
    {
        "name": "GymnasticsEvents",
        "prompts": [
            "artistic gymnastics",
            "rhythmic gymnastics",
            "trampoline gymnastics",
            "balance beam routine",
            "floor exercise performance"
        ]
    },
    {
        "name": "CombatSports",
        "prompts": [
            "boxing match",
            "judo competition",
            "wrestling bout",
            "taekwondo fight",
            "fencing duel"
        ]
    },
    {
        "name": "TeamSports",
        "prompts": [
            "basketball game",
            "volleyball match",
            "football (soccer) match",
            "handball game",
            "field hockey competition"
        ]
    },
    {
        "name": "CyclingSports",
        "prompts": [
            "road cycling race",
            "track cycling event",
            "mountain bike competition",
            "BMX racing",
            "cycling time trial"
        ]
    },
    {
        "name": "RacquetSports",
        "prompts": [
            "tennis match",
            "badminton game",
            "table tennis competition",
            "squash game",
            "tennis doubles match"
        ]
    },
    {
        "name": "RowingAndSailing",
        "prompts": [
            "rowing competition",
            "sailing race",
            "canoe sprint",
            "kayak event",
            "windsurfing competition"
        ]
    }
]
3 - Utility Function to Classify All Videos from a Particular Index
# Utility function
def print_page(page):
    for data in page:
        print(f"video_id={data.video_id}")
        for cl in data.classes:
            print(
                f"  name={cl.name} score={cl.score} duration_ratio={cl.duration_ratio} clips={cl.clips.model_dump_json(indent=2)}"
            )


result = client.classify.index(
    index_id=INDEX_ID,
    options=["visual"],
    classes=CLASSES,
    include_clips=True
)

The client.classify.index() method is called to perform the classification. It uses the previously defined INDEX_ID, focuses on visual classification (options=["visual"]), and includes clip information in the results. The CLASSES parameter specifies the classes to be used for classification.

The print_page(result) function displays the classification results in a readable format. It iterates through the classification response, printing the video ID and detailed information about each classified class, including its name, score, duration ratio, and associated clips. This approach enables easy analysis of the classification output across all videos in the specified index.

‍

4 - Function to Print the Result in a Structured and Clear Way

The print_classification_result() function displays TwelveLabs API classification results in a structured, readable format. It processes each video in the result set, presenting key information such as video ID, detected classes, and associated scores. For each class, it shows the overall score and duration ratio, followed by details of the top five clips, sorted by score in descending order. These clip details include start and end times, as well as the associated prompt in the CLASSES. The function also notes if there are additional clips beyond the top five.

def print_classification_result(result) -> None:

    for video_data in result.data:
        print(f"Video ID: {video_data.video_id}")
        print("=" * 50)


        for class_data in video_data.classes:
            print(f"  Class: {class_data.name}")
            print(f"  Score: {class_data.score:.2f}")
            print(f"  Duration Ratio: {class_data.duration_ratio:.2f}")
            print("  Clips:")


            sorted_clips = sorted(class_data.clips, key=lambda x: x.score, reverse=True)


            for i, clip in enumerate(sorted_clips[:5], 1):  # Print top 5 clips
                print(f"    {i}. Score: {clip.score:.2f}")
                print(f"       Start: {clip.start:.2f}s, End: {clip.end:.2f}s")
                print(f"       Prompt: {clip.prompt}")


            if len(sorted_clips) > 5:
                print(f"    ... and {len(sorted_clips) - 5} more clips")


            print("-" * 40)


        print("\n")


    print(f"Total results: {result.page_info.total_results}")
    print(f"Page expires at: {result.page_info.page_expired_at}")
    print(f"Next page token: {result.page_info.next_page_token}")


The sample output for the above result is -

Video ID: 66c9b03be53394f4aaed82c1
==================================================
  Class: AquaticSports
  Score: 96.08
  Duration Ratio: 0.90
  Clips:
    1. Score: 85.74
       Start: 19.30s, End: 42.00s
       Prompt: water polo match
    2. Score: 85.38
       Start: 56.30s, End: 160.41s
       Prompt: water polo match
    3. Score: 85.25
       Start: 19.30s, End: 124.07s
       Prompt: synchronized swimming
    4. Score: 85.13
       Start: 0.00s, End: 24.83s
       Prompt: swimming competition
    5. Score: 85.08
       Start: 124.10s, End: 160.41s
       Prompt: synchronized swimming
    ... and 19 more clips
5 - Specific Class Categorization from All Videos in Index

This section focuses on the specific class categorization for "AquaticSports" across all videos in a given index. The process involves three main steps: defining a specific class with associated prompts, performing the classification, and displaying the results.

This targeted approach enables more accurate classification of videos containing aquatic sports content. The print_page() function serves as a utility tool, providing a straightforward output of the classification results.

CLASS = [
    {
        "name": "AquaticSports",
        "prompts": [
            "swimming competition",
            "diving event",
            "water polo match",
            "synchronized swimming",
            "open water swimming"
        ]
    }


]


def print_page(page):
    for data in page:
        print(f"video_id={data.video_id}")
        for cl in data.classes:
            print(
                f"  name={cl.name} score={cl.score} duration_ratio={cl.duration_ratio} detailed_scores={cl.detailed_scores.model_dump_json(indent=2)}"
            )
result = client.classify.index(
    index_id=INDEX_ID,
    options=["visual"],
    classes=CLASS,
    include_clips=True,
    show_detailed_score=True
)


print_classification_result(result)

The sample output section of the above snippet would provide the videos that fall under the specific CLASS.

Video ID: 66c9b03be53394f4aaed82c1
==================================================
  Class: AquaticSports
  Score: 96.08
  Duration Ratio: 0.90
  Clips:
    1. Score: 85.74
       Start: 19.30s, End: 42.00s
       Prompt: water polo match
    2. Score: 85.38
       Start: 56.30s, End: 160.41s
       Prompt: water polo match
    3. Score: 85.25
       Start: 19.30s, End: 124.07s
       Prompt: synchronized swimming
    4. Score: 85.13
       Start: 0.00s, End: 24.83s
       Prompt: swimming competition
    5. Score: 85.08
       Start: 124.10s, End: 160.41s
       Prompt: synchronized swimming
    ... and 19 more clips

Now that you know how to implement and interact with the classify endpoint using the Twelve Labs SDK, we can move forward with developing the full Streamlit application for users.

‍

Creating the Streamlit Application

We chose to build the application on Streamlit due to its simplicity and rapid prototyping capabilities, making it ideal for the initial stages of development. Streamlit allows us to quickly create interactive web interfaces with minimal code, perfect for showcasing classification task results in a user-friendly manner. By leveraging the Twelve Labs SDK, we can create highly interactive web interfaces in minutes.

To get started with Streamlit, ensure you've set up your virtual environment and install Streamlit using the command:

pip install streamlit

For more information, check the official Streamlit documentation.

To build the Streamlit application, use the following file structure:

.
├── .env
├── app.py
├── requirements.txt
└── .gitignore

The requirements.txt file contains:

streamlit
twelvelabs
requests
python-dotenv

These dependencies cover the core functionality needed for the Olympic video clip classification application.

The main app.py file contains the following functionality:

  • get_initial_classes(): Sets up predefined Olympic sports categories.

  • get_custom_classes() and add_custom_class(): Manage user-defined categories to provide more flexibility.

  • classify_videos(): Interacts with the TwelveLabs API to perform video classification based on selected categories.

  • get_video_urls(): Retrieves video URLs for classified content.

  • render_video(): Creates an embedded video player using HLS.js for smooth playback.

# Import Necessary Dependencies
import streamlit as st
from twelvelabs import TwelveLabs
import requests
import os
from dotenv import load_dotenv

load_dotenv()

# Get the API Key from the Dashboard - https://playground.twelvelabs.io/dashboard/api-key
API_KEY = os.getenv("API_KEY")

# Create the INDEX ID as specified in the README.md and get the INDEX_ID
INDEX_ID = os.getenv("INDEX_ID")

client = TwelveLabs(api_key=API_KEY)

# Background Setting of the Application
page_element = """
<style>
[data-testid="stAppViewContainer"] {
    background-image: url("https://wallpapercave.com/wp/wp3589963.jpg");
    background-size: cover;
}
[data-testid="stHeader"] {
    background-color: rgba(0,0,0,0);
}
[data-testid="stToolbar"] {
    right: 2rem;
    background-image: url("");
    background-size: cover;
}
</style>
"""
st.markdown(page_element, unsafe_allow_html=True)


# Classes to classify the video into, there are categories name and 
# the prompts which specifc finds that factor to label that category

@st.cache_data
def get_initial_classes():
    return [
        {"name": "AquaticSports", "prompts": ["swimming competition", "diving event", "water polo match", "synchronized swimming", "open water swimming"]},
        {"name": "AthleticEvents", "prompts": ["track and field", "marathon running", "long jump competition", "javelin throw", "high jump event"]},
        {"name": "GymnasticsEvents", "prompts": ["artistic gymnastics", "rhythmic gymnastics", "trampoline gymnastics", "balance beam routine", "floor exercise performance"]},
        {"name": "CombatSports", "prompts": ["boxing match", "judo competition", "wrestling bout", "taekwondo fight", "fencing duel"]},
        {"name": "TeamSports", "prompts": ["basketball game", "volleyball match", "football (soccer) match", "handball game", "field hockey competition"]},
        {"name": "CyclingSports", "prompts": ["road cycling race", "track cycling event", "mountain bike competition", "BMX racing", "cycling time trial"]},
        {"name": "RacquetSports", "prompts": ["tennis match", "badminton game", "table tennis competition", "squash game", "tennis doubles match"]},
        {"name": "RowingAndSailing", "prompts": ["rowing competition", "sailing race", "canoe sprint", "kayak event", "windsurfing competition"]}
    ]

# Session State for the custom classes 
def get_custom_classes():
    if 'custom_classes' not in st.session_state:
        st.session_state.custom_classes = []
    return st.session_state.custom_classes

# Utitlity Function to add the custom classes in app
def add_custom_class(name, prompts):
    custom_classes = get_custom_classes()
    custom_classes.append({"name": name, "prompts": prompts})
    st.session_state.custom_classes = custom_classes
    st.session_state.new_class_added = True

# Utitlity Function to classify all the videos in the specified Index
def classify_videos(selected_classes):
    return client.classify.index(
        index_id=INDEX_ID,
        options=["visual"],
        classes=selected_classes,
        include_clips=True
    )

# To get the video urls from the resultant video id
def get_video_urls(video_ids):
    base_url = f"https://api.twelvelabs.io/v1.2/indexes/{INDEX_ID}/videos/{{}}"
    headers = {"x-api-key": API_KEY, "Content-Type": "application/json"}
    video_urls = {}

    for video_id in video_ids:
        try:
            response = requests.get(base_url.format(video_id), headers=headers)
            response.raise_for_status()
            data = response.json()
            if 'hls' in data and 'video_url' in data['hls']:
                video_urls[video_id] = data['hls']['video_url']
            else:
                st.warning(f"No video URL found for video ID: {video_id}")
        except requests.exceptions.RequestException as e:
            st.error(f"Failed to get data for video ID: {video_id}. Error: {str(e)}")

    return video_urls

# Utitlity Function to Render the Video by the resultant video url
def render_video(video_url):
    hls_player = f"""
    <script src="https://cdn.jsdelivr.net/npm/hls.js@latest"></script>
    <div style="width: 100%; border-radius: 10px; overflow: hidden; box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);">
        <video id="video" controls style="width: 100%; height: auto;"></video>
    </div>
    <script>
      var video = document.getElementById('video');
      var videoSrc = "{video_url}";
      if (Hls.isSupported()) {{
        var hls = new Hls();
        hls.loadSource(videoSrc);
        hls.attachMedia(video);
        hls.on(Hls.Events.MANIFEST_PARSED, function() {{
          video.pause();
        }});
      }}
      else if (video.canPlayType('application/vnd.apple.mpegurl')) {{
        video.src = videoSrc;
        video.addEventListener('loadedmetadata', function() {{
          video.pause();
        }});
      }}
    </script>
    """
    st.components.v1.html(hls_player, height=300)

The get_initial_classes() function returns a static list of predefined categories for Olympic sports. Streamlit caches this data, ensuring the function is executed only once and its result is saved in memory. Subsequent calls to this function return the cached result, avoiding repeated executions.

The get_initial_classes() function is cached, while get_custom_classes() is not. Custom classes, stored in Streamlit's session state, are expected to change during the application's runtime.

get_video_urls() takes a list of video IDs and makes API calls to TwelveLabs to retrieve corresponding HLS (HTTP Live Streaming) URLs, handling potential errors and missing URLs.

render_video() creates an HTML5 video player with HLS.js support, compatible with multiple browsers and devices. It falls back to native HLS if necessary.

In classify_videos(), user-selected classes are used to classify videos in the specified index using the TwelveLabs client. This function focuses on visual classification and includes clip information in the results.

# Main Function
def main():
    # Basic Markdown Setup for the Application
    st.markdown("""
    <style>
    .big-font {
        font-size: 40px !important;
        font-weight: bold;
        color: #000000;
        text-align: center;
        margin-bottom: 30px;
    }
    .subheader {
        font-size: 24px;
        font-weight: bold;
        color: #424242;
        margin-top: 20px;
        margin-bottom: 10px;
    }
    .stButton>button {
        width: 100%;
    }
    .video-info {
        background-color: #f0f0f0;
        border-radius: 10px;
        padding: 15px;
        margin-bottom: 20px;
    }
    .custom-box {
        background-color: #f9f9f9;
        border-radius: 10px;
        padding: 20px;
        margin-bottom: 20px;
        box-shadow: 0 2px 4px rgba(0,0,0,0.1);
    }
    .stTabs [data-baseweb="tab-list"] {
        gap: 24px;
    }
    .stTabs [data-baseweb="tab"] {
        height: 50px;
        white-space: pre-wrap;
        background-color: #f0f2f6;
        border-radius: 4px 4px 0px 0px;
        gap: 1px;
        padding-top: 10px;
        padding-bottom: 10px;
    }
    .stTabs [data-baseweb="tab-list"] button[aria-selected="true"] {
        background-color: #e8eaed;
    }
    </style>
    """, unsafe_allow_html=True)

    st.markdown('<p class="big-font">Olympics Classification w/t Twelve Labs</p>', unsafe_allow_html=True)
    
    # Updation of the classes
    CLASSES = get_initial_classes() + get_custom_classes()
    
    # Nav Tabs Creation
    tab1, tab2 = st.tabs(["Select Classes", "Add Custom Class"])
    
    with tab1:
        st.markdown('<p class="subheader">Select Classes</p>', unsafe_allow_html=True)
        with st.container():

            class_names = [cls["name"] for cls in CLASSES]
            # Multiselect option from the CLASSES
            selected_classes = st.multiselect("Choose one or more Olympic sports categories:", class_names)
            if st.button("Classify Videos", key="classify_button"):
                if selected_classes:
                    with st.spinner("Classifying videos..."):
                        selected_classes_with_prompts = [cls for cls in CLASSES if cls["name"] in selected_classes]
                        res = classify_videos(selected_classes_with_prompts)
                        
                        video_ids = [data.video_id for data in res.data]
                        # Retrieving the video urls from the resultant video which matches to the selected CLASSES
                        video_urls = get_video_urls(video_ids)
                        
                        st.markdown('<p class="subheader">Classified Videos</p>', unsafe_allow_html=True)
                        
                        # Iterating over to showcase the information for every resulatant video
                        for i, video_data in enumerate(res.data, 1):
                            video_id = video_data.video_id
                            video_url = video_urls.get(video_id, "URL not found")
                            
                            st.markdown(f"### Video {i}")
                            st.markdown('<div class="video-info">', unsafe_allow_html=True)
                            st.markdown(f"**Video ID:** {video_id}")
                            
                            for class_data in video_data.classes:
                                st.markdown(f"""
                                **Class:** {class_data.name}
                                - Score: {class_data.score:.2f}
                                - Duration Ratio: {class_data.duration_ratio:.2f}
                                """)
                            
                            if video_url != "URL not found":
                                render_video(video_url)
                            else:
                                st.warning("Video URL not available. Unable to render video.")
                            
                            st.markdown("---")
                        
                        st.success(f"Total videos classified: {len(res.data)}")
                else:
                    st.warning("Please select at least one class.")
            st.markdown('</div>', unsafe_allow_html=True)

        # Nav Tab for the addition of the Custom Classes to select from
        with tab2:
            st.markdown('<p class="subheader">Add Custom Class</p>', unsafe_allow_html=True)
            with st.container():
                custom_class_name = st.text_input("Enter custom class name")
                custom_class_prompts = st.text_input("Enter custom class prompts (comma-separated)")
                if st.button("Add Custom Class"):
                    if custom_class_name and custom_class_prompts:
                        add_custom_class(custom_class_name, custom_class_prompts.split(','))
                        st.success(f"Custom class '{custom_class_name}' added successfully!")
                        st.experimental_rerun()
                    else:
                        st.warning("Please enter both class name and prompts.")
                st.markdown('</div>', unsafe_allow_html=True)

    if st.session_state.get('new_class_added', False):
        st.session_state.new_class_added = False
        st.experimental_rerun()

if __name__ == "__main__":
    main()

The main function of the application focuses heavily on CSS and layout, as well as the correct sequence of instructions. Users can add new classification categories through the "Add Custom Class" tab under the video classification tab. The "Select Classes" tab allows users to choose from predefined and custom categories.

When users click the "Classify Videos" button, the application:

  • Retrieves the selected classes.

  • Calls the TwelveLabs API to classify videos (classify_videos() function).

  • Fetches video URLs for the classified videos.

  • Displays the classification results and renders the videos.

The final application results—

More Ideas to Experiment with the Tutorial

Understanding the working procedure and development of the application prepares you to implement your innovative ideas and make an impact in the world. Here are some use case ideas similar to the tutorial blog that you can build on:

🔍 Video Search Engine: Create a searchable database of video content, allowing users to find specific scenes or topics within large video collections.

🎥 Security Footage Analyzer: Detect and categorize specific events or behaviors in security camera footage.

💃 Dance Move Classifier: Identify and categorize different dance styles or specific moves from dance videos.

‍

Conclusion

This blog post aims to provide you with a detailed explanation of the working procedure and how the application is built using the classification task, leveraging Twelve Labs. Thank you for following along with the tutorial. We look forward to your ideas on improving the user experience and solving various problems.

Introduction

Ever dreamed of winning a medal for organizing Olympic video footage? 🥇 Well, now's your chance to stand on the podium!

The Olympics Video Clips Classification Application aims to streamline the tedious process of categorizing sports footage. Using Twelve Labs' Marengo 2.6 Embedding Model, this app swiftly classifies Olympic sports from video clips.

By analyzing on-screen text, conversations, and visuals, the application categorizes video clips with ease. This tutorial will guide you through creating a Streamlit application that revolutionizes how scholars, sports enthusiasts, and broadcasters interact with Olympic content. You'll learn to build an app that classifies videos based on user-defined categories. A detailed demonstration video of the application is provided below.

You can explore the demo of the application here: Olympics Application. You can also play with it via this Replit template.

‍

Prerequisite

  • Generate an API key by signing up at the Twelve Labs Playground.

  • Find the repository for the notebooks and this application on GitHub.

  • The Streamlit application uses Python, HTML, and JavaScript.


Working of the Application

This section outlines the application flow for classifying Olympic video clips.

The application uses a classification search engine, which has various potential uses. In this task, we'll focus on classifying and retrieving Olympic sports video clips. The first crucial step is creating an index.

  • To do this, visit the Twelve Labs Playground.

  • Create a New Index and select Marengo 2.6 (Embedding Engine), which is suitable for indexing and classification.

  • Once all sports videos are uploaded to this index, we can proceed.

The following section details how to create and implement an Index ID step by step. Here's an overview of the general application workflow:

  • Users have two options: they can either select the type of sport they want to view from a multi-select menu or add their own custom classes.

  • The selected class is then sent to the classify endpoint, along with the INDEX ID, to retrieve and classify all relevant video clips. Additional parameters, such as include_clips, are also sent to this endpoint.

  • The response includes the Video ID, class, confidence level, start and end times, thumbnail URL, and more.

  • To obtain the video URL associated with the resulting Video ID, we need to access the video info endpoint.

Preparation Steps

  • Sign up and create an Index on the Twelve Labs Playground.

  • Choose the appropriate option for classification: Marengo 2.6 (Embedding Engine). This engine excels in video search and classification, providing a robust foundation for video understanding.

  • Upload sports video clips. If needed, use the sample clips provided for this tutorial: Sports Clips.

With these steps completed, you're now ready to dive in and develop the application!

Twelvelabs_API=your_api_key_here
API_URL=your_api_url_here
INDEX_ID=your_index_id_here

Walkthrough for Classification

1 - Importing, Validating, and Setting Up the Client

We begin by importing the necessary libraries and setting up environment variables to create the correct environment. To get started:

import os
import requests
import glob
import requests
from twelvelabs import TwelveLabs
from twelvelabs.models.task import Task
import dotenv


API_KEY=os.getenv("Twelvelabs_API")
API_URL=os.getenv("API_URL")
INDEX_ID=os.getenv("INDEX_ID")

client = TwelveLabs(api_key=API_KEY)

# URL of the /indexes endpoint
INDEXES_URL = f"{API_URL}/indexes"
     
# Setting headers variables

default_header = {
    "x-api-key": API_KEY
}

The dependencies have been imported, environment variables loaded correctly, and the API client initialized with the proper credentials. The base URL for accessing indexes and default headers for API requests have also been set up.

With this foundation in place, you're now ready to proceed with more specific classification tasks, such as creating indexes.

‍

2 - Setting up the Classes

Below is the list of classes for classifying Olympic sports clips:

# Categories for the classification of the Olympics Sports

CLASSES = [
    {
        "name": "AquaticSports",
        "prompts": [
            "swimming competition",
            "diving event",
            "water polo match",
            "synchronized swimming",
            "open water swimming"
        ]
    },
    {
        "name": "AthleticEvents",
        "prompts": [
            "track and field",
            "marathon running",
            "long jump competition",
            "javelin throw",
            "high jump event"
        ]
    },
    {
        "name": "GymnasticsEvents",
        "prompts": [
            "artistic gymnastics",
            "rhythmic gymnastics",
            "trampoline gymnastics",
            "balance beam routine",
            "floor exercise performance"
        ]
    },
    {
        "name": "CombatSports",
        "prompts": [
            "boxing match",
            "judo competition",
            "wrestling bout",
            "taekwondo fight",
            "fencing duel"
        ]
    },
    {
        "name": "TeamSports",
        "prompts": [
            "basketball game",
            "volleyball match",
            "football (soccer) match",
            "handball game",
            "field hockey competition"
        ]
    },
    {
        "name": "CyclingSports",
        "prompts": [
            "road cycling race",
            "track cycling event",
            "mountain bike competition",
            "BMX racing",
            "cycling time trial"
        ]
    },
    {
        "name": "RacquetSports",
        "prompts": [
            "tennis match",
            "badminton game",
            "table tennis competition",
            "squash game",
            "tennis doubles match"
        ]
    },
    {
        "name": "RowingAndSailing",
        "prompts": [
            "rowing competition",
            "sailing race",
            "canoe sprint",
            "kayak event",
            "windsurfing competition"
        ]
    }
]
3 - Utility Function to Classify All Videos from a Particular Index
# Utility function
def print_page(page):
    for data in page:
        print(f"video_id={data.video_id}")
        for cl in data.classes:
            print(
                f"  name={cl.name} score={cl.score} duration_ratio={cl.duration_ratio} clips={cl.clips.model_dump_json(indent=2)}"
            )


result = client.classify.index(
    index_id=INDEX_ID,
    options=["visual"],
    classes=CLASSES,
    include_clips=True
)

The client.classify.index() method is called to perform the classification. It uses the previously defined INDEX_ID, focuses on visual classification (options=["visual"]), and includes clip information in the results. The CLASSES parameter specifies the classes to be used for classification.

The print_page(result) function displays the classification results in a readable format. It iterates through the classification response, printing the video ID and detailed information about each classified class, including its name, score, duration ratio, and associated clips. This approach enables easy analysis of the classification output across all videos in the specified index.

‍

4 - Function to Print the Result in a Structured and Clear Way

The print_classification_result() function displays TwelveLabs API classification results in a structured, readable format. It processes each video in the result set, presenting key information such as video ID, detected classes, and associated scores. For each class, it shows the overall score and duration ratio, followed by details of the top five clips, sorted by score in descending order. These clip details include start and end times, as well as the associated prompt in the CLASSES. The function also notes if there are additional clips beyond the top five.

def print_classification_result(result) -> None:

    for video_data in result.data:
        print(f"Video ID: {video_data.video_id}")
        print("=" * 50)


        for class_data in video_data.classes:
            print(f"  Class: {class_data.name}")
            print(f"  Score: {class_data.score:.2f}")
            print(f"  Duration Ratio: {class_data.duration_ratio:.2f}")
            print("  Clips:")


            sorted_clips = sorted(class_data.clips, key=lambda x: x.score, reverse=True)


            for i, clip in enumerate(sorted_clips[:5], 1):  # Print top 5 clips
                print(f"    {i}. Score: {clip.score:.2f}")
                print(f"       Start: {clip.start:.2f}s, End: {clip.end:.2f}s")
                print(f"       Prompt: {clip.prompt}")


            if len(sorted_clips) > 5:
                print(f"    ... and {len(sorted_clips) - 5} more clips")


            print("-" * 40)


        print("\n")


    print(f"Total results: {result.page_info.total_results}")
    print(f"Page expires at: {result.page_info.page_expired_at}")
    print(f"Next page token: {result.page_info.next_page_token}")


The sample output for the above result is -

Video ID: 66c9b03be53394f4aaed82c1
==================================================
  Class: AquaticSports
  Score: 96.08
  Duration Ratio: 0.90
  Clips:
    1. Score: 85.74
       Start: 19.30s, End: 42.00s
       Prompt: water polo match
    2. Score: 85.38
       Start: 56.30s, End: 160.41s
       Prompt: water polo match
    3. Score: 85.25
       Start: 19.30s, End: 124.07s
       Prompt: synchronized swimming
    4. Score: 85.13
       Start: 0.00s, End: 24.83s
       Prompt: swimming competition
    5. Score: 85.08
       Start: 124.10s, End: 160.41s
       Prompt: synchronized swimming
    ... and 19 more clips
5 - Specific Class Categorization from All Videos in Index

This section focuses on the specific class categorization for "AquaticSports" across all videos in a given index. The process involves three main steps: defining a specific class with associated prompts, performing the classification, and displaying the results.

This targeted approach enables more accurate classification of videos containing aquatic sports content. The print_page() function serves as a utility tool, providing a straightforward output of the classification results.

CLASS = [
    {
        "name": "AquaticSports",
        "prompts": [
            "swimming competition",
            "diving event",
            "water polo match",
            "synchronized swimming",
            "open water swimming"
        ]
    }


]


def print_page(page):
    for data in page:
        print(f"video_id={data.video_id}")
        for cl in data.classes:
            print(
                f"  name={cl.name} score={cl.score} duration_ratio={cl.duration_ratio} detailed_scores={cl.detailed_scores.model_dump_json(indent=2)}"
            )
result = client.classify.index(
    index_id=INDEX_ID,
    options=["visual"],
    classes=CLASS,
    include_clips=True,
    show_detailed_score=True
)


print_classification_result(result)

The sample output section of the above snippet would provide the videos that fall under the specific CLASS.

Video ID: 66c9b03be53394f4aaed82c1
==================================================
  Class: AquaticSports
  Score: 96.08
  Duration Ratio: 0.90
  Clips:
    1. Score: 85.74
       Start: 19.30s, End: 42.00s
       Prompt: water polo match
    2. Score: 85.38
       Start: 56.30s, End: 160.41s
       Prompt: water polo match
    3. Score: 85.25
       Start: 19.30s, End: 124.07s
       Prompt: synchronized swimming
    4. Score: 85.13
       Start: 0.00s, End: 24.83s
       Prompt: swimming competition
    5. Score: 85.08
       Start: 124.10s, End: 160.41s
       Prompt: synchronized swimming
    ... and 19 more clips

Now that you know how to implement and interact with the classify endpoint using the Twelve Labs SDK, we can move forward with developing the full Streamlit application for users.

‍

Creating the Streamlit Application

We chose to build the application on Streamlit due to its simplicity and rapid prototyping capabilities, making it ideal for the initial stages of development. Streamlit allows us to quickly create interactive web interfaces with minimal code, perfect for showcasing classification task results in a user-friendly manner. By leveraging the Twelve Labs SDK, we can create highly interactive web interfaces in minutes.

To get started with Streamlit, ensure you've set up your virtual environment and install Streamlit using the command:

pip install streamlit

For more information, check the official Streamlit documentation.

To build the Streamlit application, use the following file structure:

.
├── .env
├── app.py
├── requirements.txt
└── .gitignore

The requirements.txt file contains:

streamlit
twelvelabs
requests
python-dotenv

These dependencies cover the core functionality needed for the Olympic video clip classification application.

The main app.py file contains the following functionality:

  • get_initial_classes(): Sets up predefined Olympic sports categories.

  • get_custom_classes() and add_custom_class(): Manage user-defined categories to provide more flexibility.

  • classify_videos(): Interacts with the TwelveLabs API to perform video classification based on selected categories.

  • get_video_urls(): Retrieves video URLs for classified content.

  • render_video(): Creates an embedded video player using HLS.js for smooth playback.

# Import Necessary Dependencies
import streamlit as st
from twelvelabs import TwelveLabs
import requests
import os
from dotenv import load_dotenv

load_dotenv()

# Get the API Key from the Dashboard - https://playground.twelvelabs.io/dashboard/api-key
API_KEY = os.getenv("API_KEY")

# Create the INDEX ID as specified in the README.md and get the INDEX_ID
INDEX_ID = os.getenv("INDEX_ID")

client = TwelveLabs(api_key=API_KEY)

# Background Setting of the Application
page_element = """
<style>
[data-testid="stAppViewContainer"] {
    background-image: url("https://wallpapercave.com/wp/wp3589963.jpg");
    background-size: cover;
}
[data-testid="stHeader"] {
    background-color: rgba(0,0,0,0);
}
[data-testid="stToolbar"] {
    right: 2rem;
    background-image: url("");
    background-size: cover;
}
</style>
"""
st.markdown(page_element, unsafe_allow_html=True)


# Classes to classify the video into, there are categories name and 
# the prompts which specifc finds that factor to label that category

@st.cache_data
def get_initial_classes():
    return [
        {"name": "AquaticSports", "prompts": ["swimming competition", "diving event", "water polo match", "synchronized swimming", "open water swimming"]},
        {"name": "AthleticEvents", "prompts": ["track and field", "marathon running", "long jump competition", "javelin throw", "high jump event"]},
        {"name": "GymnasticsEvents", "prompts": ["artistic gymnastics", "rhythmic gymnastics", "trampoline gymnastics", "balance beam routine", "floor exercise performance"]},
        {"name": "CombatSports", "prompts": ["boxing match", "judo competition", "wrestling bout", "taekwondo fight", "fencing duel"]},
        {"name": "TeamSports", "prompts": ["basketball game", "volleyball match", "football (soccer) match", "handball game", "field hockey competition"]},
        {"name": "CyclingSports", "prompts": ["road cycling race", "track cycling event", "mountain bike competition", "BMX racing", "cycling time trial"]},
        {"name": "RacquetSports", "prompts": ["tennis match", "badminton game", "table tennis competition", "squash game", "tennis doubles match"]},
        {"name": "RowingAndSailing", "prompts": ["rowing competition", "sailing race", "canoe sprint", "kayak event", "windsurfing competition"]}
    ]

# Session State for the custom classes 
def get_custom_classes():
    if 'custom_classes' not in st.session_state:
        st.session_state.custom_classes = []
    return st.session_state.custom_classes

# Utitlity Function to add the custom classes in app
def add_custom_class(name, prompts):
    custom_classes = get_custom_classes()
    custom_classes.append({"name": name, "prompts": prompts})
    st.session_state.custom_classes = custom_classes
    st.session_state.new_class_added = True

# Utitlity Function to classify all the videos in the specified Index
def classify_videos(selected_classes):
    return client.classify.index(
        index_id=INDEX_ID,
        options=["visual"],
        classes=selected_classes,
        include_clips=True
    )

# To get the video urls from the resultant video id
def get_video_urls(video_ids):
    base_url = f"https://api.twelvelabs.io/v1.2/indexes/{INDEX_ID}/videos/{{}}"
    headers = {"x-api-key": API_KEY, "Content-Type": "application/json"}
    video_urls = {}

    for video_id in video_ids:
        try:
            response = requests.get(base_url.format(video_id), headers=headers)
            response.raise_for_status()
            data = response.json()
            if 'hls' in data and 'video_url' in data['hls']:
                video_urls[video_id] = data['hls']['video_url']
            else:
                st.warning(f"No video URL found for video ID: {video_id}")
        except requests.exceptions.RequestException as e:
            st.error(f"Failed to get data for video ID: {video_id}. Error: {str(e)}")

    return video_urls

# Utitlity Function to Render the Video by the resultant video url
def render_video(video_url):
    hls_player = f"""
    <script src="https://cdn.jsdelivr.net/npm/hls.js@latest"></script>
    <div style="width: 100%; border-radius: 10px; overflow: hidden; box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);">
        <video id="video" controls style="width: 100%; height: auto;"></video>
    </div>
    <script>
      var video = document.getElementById('video');
      var videoSrc = "{video_url}";
      if (Hls.isSupported()) {{
        var hls = new Hls();
        hls.loadSource(videoSrc);
        hls.attachMedia(video);
        hls.on(Hls.Events.MANIFEST_PARSED, function() {{
          video.pause();
        }});
      }}
      else if (video.canPlayType('application/vnd.apple.mpegurl')) {{
        video.src = videoSrc;
        video.addEventListener('loadedmetadata', function() {{
          video.pause();
        }});
      }}
    </script>
    """
    st.components.v1.html(hls_player, height=300)

The get_initial_classes() function returns a static list of predefined categories for Olympic sports. Streamlit caches this data, ensuring the function is executed only once and its result is saved in memory. Subsequent calls to this function return the cached result, avoiding repeated executions.

The get_initial_classes() function is cached, while get_custom_classes() is not. Custom classes, stored in Streamlit's session state, are expected to change during the application's runtime.

get_video_urls() takes a list of video IDs and makes API calls to TwelveLabs to retrieve corresponding HLS (HTTP Live Streaming) URLs, handling potential errors and missing URLs.

render_video() creates an HTML5 video player with HLS.js support, compatible with multiple browsers and devices. It falls back to native HLS if necessary.

In classify_videos(), user-selected classes are used to classify videos in the specified index using the TwelveLabs client. This function focuses on visual classification and includes clip information in the results.

# Main Function
def main():
    # Basic Markdown Setup for the Application
    st.markdown("""
    <style>
    .big-font {
        font-size: 40px !important;
        font-weight: bold;
        color: #000000;
        text-align: center;
        margin-bottom: 30px;
    }
    .subheader {
        font-size: 24px;
        font-weight: bold;
        color: #424242;
        margin-top: 20px;
        margin-bottom: 10px;
    }
    .stButton>button {
        width: 100%;
    }
    .video-info {
        background-color: #f0f0f0;
        border-radius: 10px;
        padding: 15px;
        margin-bottom: 20px;
    }
    .custom-box {
        background-color: #f9f9f9;
        border-radius: 10px;
        padding: 20px;
        margin-bottom: 20px;
        box-shadow: 0 2px 4px rgba(0,0,0,0.1);
    }
    .stTabs [data-baseweb="tab-list"] {
        gap: 24px;
    }
    .stTabs [data-baseweb="tab"] {
        height: 50px;
        white-space: pre-wrap;
        background-color: #f0f2f6;
        border-radius: 4px 4px 0px 0px;
        gap: 1px;
        padding-top: 10px;
        padding-bottom: 10px;
    }
    .stTabs [data-baseweb="tab-list"] button[aria-selected="true"] {
        background-color: #e8eaed;
    }
    </style>
    """, unsafe_allow_html=True)

    st.markdown('<p class="big-font">Olympics Classification w/t Twelve Labs</p>', unsafe_allow_html=True)
    
    # Updation of the classes
    CLASSES = get_initial_classes() + get_custom_classes()
    
    # Nav Tabs Creation
    tab1, tab2 = st.tabs(["Select Classes", "Add Custom Class"])
    
    with tab1:
        st.markdown('<p class="subheader">Select Classes</p>', unsafe_allow_html=True)
        with st.container():

            class_names = [cls["name"] for cls in CLASSES]
            # Multiselect option from the CLASSES
            selected_classes = st.multiselect("Choose one or more Olympic sports categories:", class_names)
            if st.button("Classify Videos", key="classify_button"):
                if selected_classes:
                    with st.spinner("Classifying videos..."):
                        selected_classes_with_prompts = [cls for cls in CLASSES if cls["name"] in selected_classes]
                        res = classify_videos(selected_classes_with_prompts)
                        
                        video_ids = [data.video_id for data in res.data]
                        # Retrieving the video urls from the resultant video which matches to the selected CLASSES
                        video_urls = get_video_urls(video_ids)
                        
                        st.markdown('<p class="subheader">Classified Videos</p>', unsafe_allow_html=True)
                        
                        # Iterating over to showcase the information for every resulatant video
                        for i, video_data in enumerate(res.data, 1):
                            video_id = video_data.video_id
                            video_url = video_urls.get(video_id, "URL not found")
                            
                            st.markdown(f"### Video {i}")
                            st.markdown('<div class="video-info">', unsafe_allow_html=True)
                            st.markdown(f"**Video ID:** {video_id}")
                            
                            for class_data in video_data.classes:
                                st.markdown(f"""
                                **Class:** {class_data.name}
                                - Score: {class_data.score:.2f}
                                - Duration Ratio: {class_data.duration_ratio:.2f}
                                """)
                            
                            if video_url != "URL not found":
                                render_video(video_url)
                            else:
                                st.warning("Video URL not available. Unable to render video.")
                            
                            st.markdown("---")
                        
                        st.success(f"Total videos classified: {len(res.data)}")
                else:
                    st.warning("Please select at least one class.")
            st.markdown('</div>', unsafe_allow_html=True)

        # Nav Tab for the addition of the Custom Classes to select from
        with tab2:
            st.markdown('<p class="subheader">Add Custom Class</p>', unsafe_allow_html=True)
            with st.container():
                custom_class_name = st.text_input("Enter custom class name")
                custom_class_prompts = st.text_input("Enter custom class prompts (comma-separated)")
                if st.button("Add Custom Class"):
                    if custom_class_name and custom_class_prompts:
                        add_custom_class(custom_class_name, custom_class_prompts.split(','))
                        st.success(f"Custom class '{custom_class_name}' added successfully!")
                        st.experimental_rerun()
                    else:
                        st.warning("Please enter both class name and prompts.")
                st.markdown('</div>', unsafe_allow_html=True)

    if st.session_state.get('new_class_added', False):
        st.session_state.new_class_added = False
        st.experimental_rerun()

if __name__ == "__main__":
    main()

The main function of the application focuses heavily on CSS and layout, as well as the correct sequence of instructions. Users can add new classification categories through the "Add Custom Class" tab under the video classification tab. The "Select Classes" tab allows users to choose from predefined and custom categories.

When users click the "Classify Videos" button, the application:

  • Retrieves the selected classes.

  • Calls the TwelveLabs API to classify videos (classify_videos() function).

  • Fetches video URLs for the classified videos.

  • Displays the classification results and renders the videos.

The final application results—

More Ideas to Experiment with the Tutorial

Understanding the working procedure and development of the application prepares you to implement your innovative ideas and make an impact in the world. Here are some use case ideas similar to the tutorial blog that you can build on:

🔍 Video Search Engine: Create a searchable database of video content, allowing users to find specific scenes or topics within large video collections.

🎥 Security Footage Analyzer: Detect and categorize specific events or behaviors in security camera footage.

💃 Dance Move Classifier: Identify and categorize different dance styles or specific moves from dance videos.

‍

Conclusion

This blog post aims to provide you with a detailed explanation of the working procedure and how the application is built using the classification task, leveraging Twelve Labs. Thank you for following along with the tutorial. We look forward to your ideas on improving the user experience and solving various problems.

Introduction

Ever dreamed of winning a medal for organizing Olympic video footage? 🥇 Well, now's your chance to stand on the podium!

The Olympics Video Clips Classification Application aims to streamline the tedious process of categorizing sports footage. Using Twelve Labs' Marengo 2.6 Embedding Model, this app swiftly classifies Olympic sports from video clips.

By analyzing on-screen text, conversations, and visuals, the application categorizes video clips with ease. This tutorial will guide you through creating a Streamlit application that revolutionizes how scholars, sports enthusiasts, and broadcasters interact with Olympic content. You'll learn to build an app that classifies videos based on user-defined categories. A detailed demonstration video of the application is provided below.

You can explore the demo of the application here: Olympics Application. You can also play with it via this Replit template.

‍

Prerequisite

  • Generate an API key by signing up at the Twelve Labs Playground.

  • Find the repository for the notebooks and this application on GitHub.

  • The Streamlit application uses Python, HTML, and JavaScript.


Working of the Application

This section outlines the application flow for classifying Olympic video clips.

The application uses a classification search engine, which has various potential uses. In this task, we'll focus on classifying and retrieving Olympic sports video clips. The first crucial step is creating an index.

  • To do this, visit the Twelve Labs Playground.

  • Create a New Index and select Marengo 2.6 (Embedding Engine), which is suitable for indexing and classification.

  • Once all sports videos are uploaded to this index, we can proceed.

The following section details how to create and implement an Index ID step by step. Here's an overview of the general application workflow:

  • Users have two options: they can either select the type of sport they want to view from a multi-select menu or add their own custom classes.

  • The selected class is then sent to the classify endpoint, along with the INDEX ID, to retrieve and classify all relevant video clips. Additional parameters, such as include_clips, are also sent to this endpoint.

  • The response includes the Video ID, class, confidence level, start and end times, thumbnail URL, and more.

  • To obtain the video URL associated with the resulting Video ID, we need to access the video info endpoint.

Preparation Steps

  • Sign up and create an Index on the Twelve Labs Playground.

  • Choose the appropriate option for classification: Marengo 2.6 (Embedding Engine). This engine excels in video search and classification, providing a robust foundation for video understanding.

  • Upload sports video clips. If needed, use the sample clips provided for this tutorial: Sports Clips.

With these steps completed, you're now ready to dive in and develop the application!

Twelvelabs_API=your_api_key_here
API_URL=your_api_url_here
INDEX_ID=your_index_id_here

Walkthrough for Classification

1 - Importing, Validating, and Setting Up the Client

We begin by importing the necessary libraries and setting up environment variables to create the correct environment. To get started:

import os
import requests
import glob
import requests
from twelvelabs import TwelveLabs
from twelvelabs.models.task import Task
import dotenv


API_KEY=os.getenv("Twelvelabs_API")
API_URL=os.getenv("API_URL")
INDEX_ID=os.getenv("INDEX_ID")

client = TwelveLabs(api_key=API_KEY)

# URL of the /indexes endpoint
INDEXES_URL = f"{API_URL}/indexes"
     
# Setting headers variables

default_header = {
    "x-api-key": API_KEY
}

The dependencies have been imported, environment variables loaded correctly, and the API client initialized with the proper credentials. The base URL for accessing indexes and default headers for API requests have also been set up.

With this foundation in place, you're now ready to proceed with more specific classification tasks, such as creating indexes.

‍

2 - Setting up the Classes

Below is the list of classes for classifying Olympic sports clips:

# Categories for the classification of the Olympics Sports

CLASSES = [
    {
        "name": "AquaticSports",
        "prompts": [
            "swimming competition",
            "diving event",
            "water polo match",
            "synchronized swimming",
            "open water swimming"
        ]
    },
    {
        "name": "AthleticEvents",
        "prompts": [
            "track and field",
            "marathon running",
            "long jump competition",
            "javelin throw",
            "high jump event"
        ]
    },
    {
        "name": "GymnasticsEvents",
        "prompts": [
            "artistic gymnastics",
            "rhythmic gymnastics",
            "trampoline gymnastics",
            "balance beam routine",
            "floor exercise performance"
        ]
    },
    {
        "name": "CombatSports",
        "prompts": [
            "boxing match",
            "judo competition",
            "wrestling bout",
            "taekwondo fight",
            "fencing duel"
        ]
    },
    {
        "name": "TeamSports",
        "prompts": [
            "basketball game",
            "volleyball match",
            "football (soccer) match",
            "handball game",
            "field hockey competition"
        ]
    },
    {
        "name": "CyclingSports",
        "prompts": [
            "road cycling race",
            "track cycling event",
            "mountain bike competition",
            "BMX racing",
            "cycling time trial"
        ]
    },
    {
        "name": "RacquetSports",
        "prompts": [
            "tennis match",
            "badminton game",
            "table tennis competition",
            "squash game",
            "tennis doubles match"
        ]
    },
    {
        "name": "RowingAndSailing",
        "prompts": [
            "rowing competition",
            "sailing race",
            "canoe sprint",
            "kayak event",
            "windsurfing competition"
        ]
    }
]
3 - Utility Function to Classify All Videos from a Particular Index
# Utility function
def print_page(page):
    for data in page:
        print(f"video_id={data.video_id}")
        for cl in data.classes:
            print(
                f"  name={cl.name} score={cl.score} duration_ratio={cl.duration_ratio} clips={cl.clips.model_dump_json(indent=2)}"
            )


result = client.classify.index(
    index_id=INDEX_ID,
    options=["visual"],
    classes=CLASSES,
    include_clips=True
)

The client.classify.index() method is called to perform the classification. It uses the previously defined INDEX_ID, focuses on visual classification (options=["visual"]), and includes clip information in the results. The CLASSES parameter specifies the classes to be used for classification.

The print_page(result) function displays the classification results in a readable format. It iterates through the classification response, printing the video ID and detailed information about each classified class, including its name, score, duration ratio, and associated clips. This approach enables easy analysis of the classification output across all videos in the specified index.

‍

4 - Function to Print the Result in a Structured and Clear Way

The print_classification_result() function displays TwelveLabs API classification results in a structured, readable format. It processes each video in the result set, presenting key information such as video ID, detected classes, and associated scores. For each class, it shows the overall score and duration ratio, followed by details of the top five clips, sorted by score in descending order. These clip details include start and end times, as well as the associated prompt in the CLASSES. The function also notes if there are additional clips beyond the top five.

def print_classification_result(result) -> None:

    for video_data in result.data:
        print(f"Video ID: {video_data.video_id}")
        print("=" * 50)


        for class_data in video_data.classes:
            print(f"  Class: {class_data.name}")
            print(f"  Score: {class_data.score:.2f}")
            print(f"  Duration Ratio: {class_data.duration_ratio:.2f}")
            print("  Clips:")


            sorted_clips = sorted(class_data.clips, key=lambda x: x.score, reverse=True)


            for i, clip in enumerate(sorted_clips[:5], 1):  # Print top 5 clips
                print(f"    {i}. Score: {clip.score:.2f}")
                print(f"       Start: {clip.start:.2f}s, End: {clip.end:.2f}s")
                print(f"       Prompt: {clip.prompt}")


            if len(sorted_clips) > 5:
                print(f"    ... and {len(sorted_clips) - 5} more clips")


            print("-" * 40)


        print("\n")


    print(f"Total results: {result.page_info.total_results}")
    print(f"Page expires at: {result.page_info.page_expired_at}")
    print(f"Next page token: {result.page_info.next_page_token}")


The sample output for the above result is -

Video ID: 66c9b03be53394f4aaed82c1
==================================================
  Class: AquaticSports
  Score: 96.08
  Duration Ratio: 0.90
  Clips:
    1. Score: 85.74
       Start: 19.30s, End: 42.00s
       Prompt: water polo match
    2. Score: 85.38
       Start: 56.30s, End: 160.41s
       Prompt: water polo match
    3. Score: 85.25
       Start: 19.30s, End: 124.07s
       Prompt: synchronized swimming
    4. Score: 85.13
       Start: 0.00s, End: 24.83s
       Prompt: swimming competition
    5. Score: 85.08
       Start: 124.10s, End: 160.41s
       Prompt: synchronized swimming
    ... and 19 more clips
5 - Specific Class Categorization from All Videos in Index

This section focuses on the specific class categorization for "AquaticSports" across all videos in a given index. The process involves three main steps: defining a specific class with associated prompts, performing the classification, and displaying the results.

This targeted approach enables more accurate classification of videos containing aquatic sports content. The print_page() function serves as a utility tool, providing a straightforward output of the classification results.

CLASS = [
    {
        "name": "AquaticSports",
        "prompts": [
            "swimming competition",
            "diving event",
            "water polo match",
            "synchronized swimming",
            "open water swimming"
        ]
    }


]


def print_page(page):
    for data in page:
        print(f"video_id={data.video_id}")
        for cl in data.classes:
            print(
                f"  name={cl.name} score={cl.score} duration_ratio={cl.duration_ratio} detailed_scores={cl.detailed_scores.model_dump_json(indent=2)}"
            )
result = client.classify.index(
    index_id=INDEX_ID,
    options=["visual"],
    classes=CLASS,
    include_clips=True,
    show_detailed_score=True
)


print_classification_result(result)

The sample output section of the above snippet would provide the videos that fall under the specific CLASS.

Video ID: 66c9b03be53394f4aaed82c1
==================================================
  Class: AquaticSports
  Score: 96.08
  Duration Ratio: 0.90
  Clips:
    1. Score: 85.74
       Start: 19.30s, End: 42.00s
       Prompt: water polo match
    2. Score: 85.38
       Start: 56.30s, End: 160.41s
       Prompt: water polo match
    3. Score: 85.25
       Start: 19.30s, End: 124.07s
       Prompt: synchronized swimming
    4. Score: 85.13
       Start: 0.00s, End: 24.83s
       Prompt: swimming competition
    5. Score: 85.08
       Start: 124.10s, End: 160.41s
       Prompt: synchronized swimming
    ... and 19 more clips

Now that you know how to implement and interact with the classify endpoint using the Twelve Labs SDK, we can move forward with developing the full Streamlit application for users.

‍

Creating the Streamlit Application

We chose to build the application on Streamlit due to its simplicity and rapid prototyping capabilities, making it ideal for the initial stages of development. Streamlit allows us to quickly create interactive web interfaces with minimal code, perfect for showcasing classification task results in a user-friendly manner. By leveraging the Twelve Labs SDK, we can create highly interactive web interfaces in minutes.

To get started with Streamlit, ensure you've set up your virtual environment and install Streamlit using the command:

pip install streamlit

For more information, check the official Streamlit documentation.

To build the Streamlit application, use the following file structure:

.
├── .env
├── app.py
├── requirements.txt
└── .gitignore

The requirements.txt file contains:

streamlit
twelvelabs
requests
python-dotenv

These dependencies cover the core functionality needed for the Olympic video clip classification application.

The main app.py file contains the following functionality:

  • get_initial_classes(): Sets up predefined Olympic sports categories.

  • get_custom_classes() and add_custom_class(): Manage user-defined categories to provide more flexibility.

  • classify_videos(): Interacts with the TwelveLabs API to perform video classification based on selected categories.

  • get_video_urls(): Retrieves video URLs for classified content.

  • render_video(): Creates an embedded video player using HLS.js for smooth playback.

# Import Necessary Dependencies
import streamlit as st
from twelvelabs import TwelveLabs
import requests
import os
from dotenv import load_dotenv

load_dotenv()

# Get the API Key from the Dashboard - https://playground.twelvelabs.io/dashboard/api-key
API_KEY = os.getenv("API_KEY")

# Create the INDEX ID as specified in the README.md and get the INDEX_ID
INDEX_ID = os.getenv("INDEX_ID")

client = TwelveLabs(api_key=API_KEY)

# Background Setting of the Application
page_element = """
<style>
[data-testid="stAppViewContainer"] {
    background-image: url("https://wallpapercave.com/wp/wp3589963.jpg");
    background-size: cover;
}
[data-testid="stHeader"] {
    background-color: rgba(0,0,0,0);
}
[data-testid="stToolbar"] {
    right: 2rem;
    background-image: url("");
    background-size: cover;
}
</style>
"""
st.markdown(page_element, unsafe_allow_html=True)


# Classes to classify the video into, there are categories name and 
# the prompts which specifc finds that factor to label that category

@st.cache_data
def get_initial_classes():
    return [
        {"name": "AquaticSports", "prompts": ["swimming competition", "diving event", "water polo match", "synchronized swimming", "open water swimming"]},
        {"name": "AthleticEvents", "prompts": ["track and field", "marathon running", "long jump competition", "javelin throw", "high jump event"]},
        {"name": "GymnasticsEvents", "prompts": ["artistic gymnastics", "rhythmic gymnastics", "trampoline gymnastics", "balance beam routine", "floor exercise performance"]},
        {"name": "CombatSports", "prompts": ["boxing match", "judo competition", "wrestling bout", "taekwondo fight", "fencing duel"]},
        {"name": "TeamSports", "prompts": ["basketball game", "volleyball match", "football (soccer) match", "handball game", "field hockey competition"]},
        {"name": "CyclingSports", "prompts": ["road cycling race", "track cycling event", "mountain bike competition", "BMX racing", "cycling time trial"]},
        {"name": "RacquetSports", "prompts": ["tennis match", "badminton game", "table tennis competition", "squash game", "tennis doubles match"]},
        {"name": "RowingAndSailing", "prompts": ["rowing competition", "sailing race", "canoe sprint", "kayak event", "windsurfing competition"]}
    ]

# Session State for the custom classes 
def get_custom_classes():
    if 'custom_classes' not in st.session_state:
        st.session_state.custom_classes = []
    return st.session_state.custom_classes

# Utitlity Function to add the custom classes in app
def add_custom_class(name, prompts):
    custom_classes = get_custom_classes()
    custom_classes.append({"name": name, "prompts": prompts})
    st.session_state.custom_classes = custom_classes
    st.session_state.new_class_added = True

# Utitlity Function to classify all the videos in the specified Index
def classify_videos(selected_classes):
    return client.classify.index(
        index_id=INDEX_ID,
        options=["visual"],
        classes=selected_classes,
        include_clips=True
    )

# To get the video urls from the resultant video id
def get_video_urls(video_ids):
    base_url = f"https://api.twelvelabs.io/v1.2/indexes/{INDEX_ID}/videos/{{}}"
    headers = {"x-api-key": API_KEY, "Content-Type": "application/json"}
    video_urls = {}

    for video_id in video_ids:
        try:
            response = requests.get(base_url.format(video_id), headers=headers)
            response.raise_for_status()
            data = response.json()
            if 'hls' in data and 'video_url' in data['hls']:
                video_urls[video_id] = data['hls']['video_url']
            else:
                st.warning(f"No video URL found for video ID: {video_id}")
        except requests.exceptions.RequestException as e:
            st.error(f"Failed to get data for video ID: {video_id}. Error: {str(e)}")

    return video_urls

# Utitlity Function to Render the Video by the resultant video url
def render_video(video_url):
    hls_player = f"""
    <script src="https://cdn.jsdelivr.net/npm/hls.js@latest"></script>
    <div style="width: 100%; border-radius: 10px; overflow: hidden; box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);">
        <video id="video" controls style="width: 100%; height: auto;"></video>
    </div>
    <script>
      var video = document.getElementById('video');
      var videoSrc = "{video_url}";
      if (Hls.isSupported()) {{
        var hls = new Hls();
        hls.loadSource(videoSrc);
        hls.attachMedia(video);
        hls.on(Hls.Events.MANIFEST_PARSED, function() {{
          video.pause();
        }});
      }}
      else if (video.canPlayType('application/vnd.apple.mpegurl')) {{
        video.src = videoSrc;
        video.addEventListener('loadedmetadata', function() {{
          video.pause();
        }});
      }}
    </script>
    """
    st.components.v1.html(hls_player, height=300)

The get_initial_classes() function returns a static list of predefined categories for Olympic sports. Streamlit caches this data, ensuring the function is executed only once and its result is saved in memory. Subsequent calls to this function return the cached result, avoiding repeated executions.

The get_initial_classes() function is cached, while get_custom_classes() is not. Custom classes, stored in Streamlit's session state, are expected to change during the application's runtime.

get_video_urls() takes a list of video IDs and makes API calls to TwelveLabs to retrieve corresponding HLS (HTTP Live Streaming) URLs, handling potential errors and missing URLs.

render_video() creates an HTML5 video player with HLS.js support, compatible with multiple browsers and devices. It falls back to native HLS if necessary.

In classify_videos(), user-selected classes are used to classify videos in the specified index using the TwelveLabs client. This function focuses on visual classification and includes clip information in the results.

# Main Function
def main():
    # Basic Markdown Setup for the Application
    st.markdown("""
    <style>
    .big-font {
        font-size: 40px !important;
        font-weight: bold;
        color: #000000;
        text-align: center;
        margin-bottom: 30px;
    }
    .subheader {
        font-size: 24px;
        font-weight: bold;
        color: #424242;
        margin-top: 20px;
        margin-bottom: 10px;
    }
    .stButton>button {
        width: 100%;
    }
    .video-info {
        background-color: #f0f0f0;
        border-radius: 10px;
        padding: 15px;
        margin-bottom: 20px;
    }
    .custom-box {
        background-color: #f9f9f9;
        border-radius: 10px;
        padding: 20px;
        margin-bottom: 20px;
        box-shadow: 0 2px 4px rgba(0,0,0,0.1);
    }
    .stTabs [data-baseweb="tab-list"] {
        gap: 24px;
    }
    .stTabs [data-baseweb="tab"] {
        height: 50px;
        white-space: pre-wrap;
        background-color: #f0f2f6;
        border-radius: 4px 4px 0px 0px;
        gap: 1px;
        padding-top: 10px;
        padding-bottom: 10px;
    }
    .stTabs [data-baseweb="tab-list"] button[aria-selected="true"] {
        background-color: #e8eaed;
    }
    </style>
    """, unsafe_allow_html=True)

    st.markdown('<p class="big-font">Olympics Classification w/t Twelve Labs</p>', unsafe_allow_html=True)
    
    # Updation of the classes
    CLASSES = get_initial_classes() + get_custom_classes()
    
    # Nav Tabs Creation
    tab1, tab2 = st.tabs(["Select Classes", "Add Custom Class"])
    
    with tab1:
        st.markdown('<p class="subheader">Select Classes</p>', unsafe_allow_html=True)
        with st.container():

            class_names = [cls["name"] for cls in CLASSES]
            # Multiselect option from the CLASSES
            selected_classes = st.multiselect("Choose one or more Olympic sports categories:", class_names)
            if st.button("Classify Videos", key="classify_button"):
                if selected_classes:
                    with st.spinner("Classifying videos..."):
                        selected_classes_with_prompts = [cls for cls in CLASSES if cls["name"] in selected_classes]
                        res = classify_videos(selected_classes_with_prompts)
                        
                        video_ids = [data.video_id for data in res.data]
                        # Retrieving the video urls from the resultant video which matches to the selected CLASSES
                        video_urls = get_video_urls(video_ids)
                        
                        st.markdown('<p class="subheader">Classified Videos</p>', unsafe_allow_html=True)
                        
                        # Iterating over to showcase the information for every resulatant video
                        for i, video_data in enumerate(res.data, 1):
                            video_id = video_data.video_id
                            video_url = video_urls.get(video_id, "URL not found")
                            
                            st.markdown(f"### Video {i}")
                            st.markdown('<div class="video-info">', unsafe_allow_html=True)
                            st.markdown(f"**Video ID:** {video_id}")
                            
                            for class_data in video_data.classes:
                                st.markdown(f"""
                                **Class:** {class_data.name}
                                - Score: {class_data.score:.2f}
                                - Duration Ratio: {class_data.duration_ratio:.2f}
                                """)
                            
                            if video_url != "URL not found":
                                render_video(video_url)
                            else:
                                st.warning("Video URL not available. Unable to render video.")
                            
                            st.markdown("---")
                        
                        st.success(f"Total videos classified: {len(res.data)}")
                else:
                    st.warning("Please select at least one class.")
            st.markdown('</div>', unsafe_allow_html=True)

        # Nav Tab for the addition of the Custom Classes to select from
        with tab2:
            st.markdown('<p class="subheader">Add Custom Class</p>', unsafe_allow_html=True)
            with st.container():
                custom_class_name = st.text_input("Enter custom class name")
                custom_class_prompts = st.text_input("Enter custom class prompts (comma-separated)")
                if st.button("Add Custom Class"):
                    if custom_class_name and custom_class_prompts:
                        add_custom_class(custom_class_name, custom_class_prompts.split(','))
                        st.success(f"Custom class '{custom_class_name}' added successfully!")
                        st.experimental_rerun()
                    else:
                        st.warning("Please enter both class name and prompts.")
                st.markdown('</div>', unsafe_allow_html=True)

    if st.session_state.get('new_class_added', False):
        st.session_state.new_class_added = False
        st.experimental_rerun()

if __name__ == "__main__":
    main()

The main function of the application focuses heavily on CSS and layout, as well as the correct sequence of instructions. Users can add new classification categories through the "Add Custom Class" tab under the video classification tab. The "Select Classes" tab allows users to choose from predefined and custom categories.

When users click the "Classify Videos" button, the application:

  • Retrieves the selected classes.

  • Calls the TwelveLabs API to classify videos (classify_videos() function).

  • Fetches video URLs for the classified videos.

  • Displays the classification results and renders the videos.

The final application results—

More Ideas to Experiment with the Tutorial

Understanding the working procedure and development of the application prepares you to implement your innovative ideas and make an impact in the world. Here are some use case ideas similar to the tutorial blog that you can build on:

🔍 Video Search Engine: Create a searchable database of video content, allowing users to find specific scenes or topics within large video collections.

🎥 Security Footage Analyzer: Detect and categorize specific events or behaviors in security camera footage.

💃 Dance Move Classifier: Identify and categorize different dance styles or specific moves from dance videos.

‍

Conclusion

This blog post aims to provide you with a detailed explanation of the working procedure and how the application is built using the classification task, leveraging Twelve Labs. Thank you for following along with the tutorial. We look forward to your ideas on improving the user experience and solving various problems.

Related articles

Building a Security Analysis Application with Twelve Labs

Building a Multimodal Retrieval-Augmented Generation Application with Twelve Labs and Milvus

Building a Video AI Interview Analyzer with Twelve Labs

Crop and Seek: Experience Advanced Video Search

Footer accent

Product

Search

Generate

Embed

Pricing

Social

LinkedIn

Youtube

For Developers

API Docs

SDKs

Discord

Status

Research

Enterprise

Company

About Us

Careers

Blog

Legal

Terms of Service

Privacy Policy

Footer accent

Product

Search

Generate

Embed

Pricing

Social

LinkedIn

Youtube

For Developers

API Docs

SDKs

Discord

Status

Research

Enterprise

Company

About Us

Careers

Blog

Legal

Terms of Service

Privacy Policy

Footer accent

Product

Search

Generate

Embed

Pricing

Social

LinkedIn

Youtube

For Developers

API Docs

SDKs

Discord

Status

Research

Enterprise

Company

About Us

Careers

Blog

Legal

Terms of Service

Privacy Policy