Supercharge Your Video Search with Twelve Labs and Roe AI

TLDR: This tutorial demonstrates how to integrate Twelve Labs' Embed API with Roe AI's platform to create a powerful semantic video search solution. By following these steps, developers can quickly set up a system that leverages state-of-the-art video embeddings for accurate and context-aware video search capabilities. Big thanks to the Roe AI team (Sam Liang and Richard Meng) for collaborating with us on this tutorial.

‍

Introduction

This tutorial focuses on integrating Twelve Labs' Embed API with Roe AI's SQL-based AI platform, providing developers with advanced tools for video understanding. This integration combines Twelve Labs' video embedding model with Roe AI's data management and search capabilities, enabling the creation of sophisticated video search solutions.

As a developer, you'll learn to leverage this integration to build applications that understand context, content, and nuance across multiple modalities in video data. The tutorial explores how this integration simplifies working with complex video data and improves the accuracy and efficiency of video search functionalities. Whether you're developing a content platform, a video analytics tool, or any application involving video data, you'll gain valuable insights into implementing these technologies in your projects.

Step 1: Obtain Your API Token from app.roe-ai.com

To get started with Roe AI, you need to obtain your API token, which will allow you to authenticate your requests. Follow these steps to retrieve your API token:

1. Sign Up or Log In: Visit app.roe-ai.com and either sign up for a new account or log in if you already have one.
2. Access API Settings: Once logged in, navigate to the API settings or your account settings where you can find the option to generate or view your API token.
3. Copy Your API Token: After generating the token, copy it to your clipboard. Make sure to keep this token secure, as it will be used in your API requests.
4. Integrate the Token in Your Code: Use the following code snippet to set up your headers for making API requests. Replace <Your API Token> with the token you copied.

import requests
import json

headers = {
  'Authorization': 'Bearer <Your API Token>'
}

base_api_url = "https://api.roe-ai.com/v1"

Now you are ready to use the Roe AI API with your token for further steps in the workflow!

‍

Step 2: Create a Roe Dataset to Store Your Videos

To store your videos in Roe AI, you need to create a dataset. Follow these steps to create a new dataset:

1. Define Your Dataset Name: Choose a name for your dataset that will help you identify it later. For this example, we will use "My Video Dataset". You can replace it with any name of your choice.
2. Set Up the API Request: Use the following code snippet to create the dataset. Ensure that you have already set up your headers and base API URL from Step 1.

create_dataset_url = f"{base_api_url}/datasets/"

dataset_name = "My Video Dataset"  # Replace with your dataset name

form_data = {
    "name": dataset_name
}

response = requests.post(create_dataset_url, headers=headers, data=form_data)

if response.status_code == 201:
    print("Success")
else:
    print("Failure!")

dataset_id = response.json().get("id") if response.status_code == 201 else None

print(f"Dataset ID: {dataset_id}")

3. Execute the Code: Run the code snippet in your Python environment.
4. Check the Response: If the dataset is created successfully, you will see "Success" printed in the console, along with the dataset ID. If there is a failure, it will print "Failure!" and the dataset ID will be None.
5. Troubleshooting: If you encounter a failure, ensure that your API token is valid and that you have the necessary permissions to create datasets in Roe AI.

Once the dataset is created successfully, you can proceed to upload your video files in the next step.

‍

‍Step 3: Upload Video Files to the Created Roe Dataset

‍Now that you have created a dataset, it's time to upload your video files. Follow these steps to upload your videos:

1. Gather Your Video Files: Ensure that you have all the video files you want to upload ready. For this example, we will assume you have a file named "video1.mp4" located at "/path/to/video1.mp4".
2. Set Up the API Request: Use the following code snippet to upload your video files to the dataset. Make sure you have the dataset ID from the previous step.

upload_files_url = f"{base_api_url}/datasets/files/upload/"

payload = {
    'dataset_id': f"{dataset_id}",
}

# REQUIRED: Add all your VIDEO FILES HERE
files = [
    ('file', ('video1.mp4', open('/path/to/video1.mp4', 'rb'), 'video/mp4')),
    # Add more files as needed
]

response = requests.request("POST", upload_files_url, headers=headers, data=payload, files=files)

if response.status_code == 200:
    print("Success")
else:
    print("Failure!")

3. Replace File Details: Replace 'video1.mp4' with the actual name of your video file and '/path/to/video1.mp4' with the correct file path on your local machine. Add more file entries in the files list for each video you want to upload.
4. Execute the Code: Run the code snippet in your Python environment.
5. Check the Response: If the upload is successful, you will see "Success" printed in the console. If there is a failure, it will print "Failure!".
6. Troubleshooting: If you encounter a failure, ensure that your API token is valid, the dataset ID is correct, and that you have the necessary permissions to upload files to the dataset.

Once the video files are uploaded successfully, you can proceed to create a table from the dataset in the next step.

‍

‍Step 4: Create a Table from the Roe Dataset

‍After uploading your video files to the dataset, the next step is to create a table from the dataset. This table will serve as the foundation for your semantic video search. Follow these steps:

1. Choose a Table Name: Select a name for your table that will help you identify it later. For this example, we will use "my_video_table". Make sure the name does not contain any hyphens (``).
2. Set Up the API Request: Use the following code snippet to create the table from the dataset. Ensure that you have the dataset ID from the previous step.

create_table_url = f"{base_api_url}/database/table/import/roe-dataset/"

table_name = "my_video_table"  # Replace with your desired table name

form_data = {
    "table_name": table_name,
    "dataset_id": f"{dataset_id}",
    "sync_dataset": "false",  # Set to "true" if you want to sync the dataset with the table
}

response = requests.post(create_table_url, headers=headers, data=form_data)

if response.status_code == 200:
    print("Success")
else:
    print("Failure!")

3. Sync Dataset Option: The sync_dataset parameter determines whether you want to keep the table synchronized with the dataset. If set to "true", any changes made to the dataset will be reflected in the table. For this example, we set it to "false".
4. Execute the Code: Run the code snippet in your Python environment.
5. Check the Response: If the table is created successfully, you will see "Success" printed in the console. If there is a failure, it will print "Failure!".
6. Troubleshooting: If you encounter a failure, ensure that your API token is valid, the dataset ID is correct, and that you have the necessary permissions to create tables in Roe AI.

Once the table is created successfully, you can proceed to get the table columns in the next step.

‍

‍Step 5: Get the Table Columns to Inspect the Available Data

‍To understand the structure of the data you have in your table, you need to retrieve the columns of the table. This will help you identify the relevant fields for your search index. Follow these steps:

1. Set Up the API Request: Use the following code snippet to get the columns from the table you created. Ensure that you have the correct table name from the previous step.

table_cols_url = f"{base_api_url}/database/table/{table_name}/columns"

response = requests.get(table_cols_url, headers=headers)

if response.status_code == 200:
    columns = response.json()  # Parsing the JSON response
    print("Success")
    print("Columns:", columns)
else:
    print("Failure! Status Code:", response.status_code)

2. Execute the Code: Run the code snippet in your Python environment.
3. Check the Response: If the request is successful, you will see "Success" printed in the console along with the list of columns available in the table. If there is a failure, it will print "Failure!" along with the status code.
4. Troubleshooting: If you encounter a failure, check the following:
   • Ensure that your API token is valid and has the necessary permissions.
   • Verify that the table name is correct and matches the one you created in the previous step.
   • If you receive a status code of 401, it indicates an unauthorized request, which typically means there is an issue with your API token.
5. Select Relevant Columns: Once you have the columns, you can identify the column named 'files', which corresponds to the video data. You can create a list of selected columns for the search index as follows:

# Assuming the columns are retrieved successfully
selected_columns = [columns[2]]  # Adjust the index based on the actual structure of the response

After successfully retrieving the table columns, you can proceed to create a search index on the table in the next step.

Step 6: Create a Search Index on the Table Using Twelve Labs' Marengo-2.6 Video Embedding Model

To enable semantic search capabilities on your video data, you need to create a search index using the Twelve Labs Marengo-2.6 embedding model. This step will configure the index to utilize various embedding models for different data types. Follow these steps:

1. Set Up the API Request: Use the following code snippet to create the search index. Ensure that you have the correct table name and selected columns from the previous steps.

create_index_url = f"{base_api_url}/index/"
search_index_config = {
    "version": '1.0',
    "pdf": {
        "text_embedder_name": "jina-clip-v1",
        "image_embedder_name": "jina-clip-v1",
    },
    "text": {
        "embedder_name": "text-embedding-3-small",
    },
    "image": {
        "embedder_name": "jina-clip-v1",
    },
    "audio": {
        "embedder_name": 'None',
    },
    "video": {
        "embedder_name": "Marengo-retrieval-2.6",
    },
}

index_name = "my_search_index"  # Replace with your desired index name

form_data = {
    "name": index_name,
    "table_name": table_name,
    "id_column_name": "r_uuid",  # Replace with the actual ID column name if different
    "column_names": json.dumps(selected_columns),
    "search_index_config": json.dumps(search_index_config),
    "display_name": index_name,
}

response = requests.post(create_index_url, headers=headers, data=form_data)

if response.status_code == 201:
    print("Success")
else:
    print("Failure!")

index_id = response.json().get("id") if response.status_code == 201 else None
print(f"Index ID: {index_id}")

2. Configure the Search Index: The search_index_config dictionary specifies the embedding models for different data types:
   • Text: Uses the "text-embedding-3-small" model for textual data.
   • Video: Uses the "Marengo-retrieval-2.6" model for video data. You can follow the docs here that explain how it works: https://docs.twelvelabs.io/docs/create-video-embeddings
   • PDF and Image: Uses "jina-clip-v1" for both.
   • Audio: Set to 'None' as audio embedding is not utilized in this case.
3. Execute the Code: Run the code snippet in your Python environment.
4. Check the Response: If the index is created successfully, you will see "Success" printed in the console. If there is a failure, it will print "Failure!" along with the status code.
5. Troubleshooting: If you encounter a failure, check the following:
   • Ensure that the API token is valid and has the necessary permissions.
   • Verify that the table name and selected columns are correct.
   • If the status code is not 201, it indicates an issue with the request, which may require checking the API documentation for any specific requirements.
6. Index ID: If the index creation is successful, the index ID will be printed, which you can use for future queries and operations.

By completing this step, you will have set up a search index that leverages the advanced capabilities of Twelve Labs' Marengo-2.6 model, enabling efficient and effective semantic searches on your video data.

‍

Step 7: Check the Status of the Search Index Creation

After initiating the creation of your search index, it's important to monitor its status to ensure that the process completes successfully. Follow these steps to check the status of the search index creation:

1. Set Up the API Request: Use the following code snippet to check the status of the search index. Make sure you have the correct index ID from the previous step.

import time

check_status_url = f"{base_api_url}/index/{index_id}/status/"

max_retries = 120  # Maximum number of retries to check the status

# Start a loop to continuously check the status
for _ in range(max_retries):
    response = requests.get(check_status_url, headers=headers)

    if response.status_code == 200:
        status = response.json().get('status')

        # Check if the index creation is complete
        if status == 'SUCCESS':
            print("Index creation succeeded.")
            break
        elif status == 'FAILURE' or status == 'REVOKED':
            print("Index creation failed.")
            break

        print(f"Index status: {status}. Checking again in 10 seconds...")
        # Wait for 10 seconds before checking again
        time.sleep(10)
    else:
        print(f"Failed to get status: {response.status_code}")
        break
else:
    print("Max retries reached, exiting.")

2. Execute the Code: Run the code snippet in your Python environment.
3. Monitor the Status: The code will check the status of the index creation up to a maximum of 120 times (or until it receives a definitive status). It will wait for 10 seconds between each check.
4. Check the Response:
   • If the status is SUCCESS, you will see "Index creation succeeded." printed in the console.
   • If the status is FAILURE or REVOKED, it will print "Index creation failed."
   • If the status is still pending, it will print the current status and continue checking.
5. Troubleshooting: If you encounter a failure in getting the status, check the following:
   • Ensure that your API token is valid and has the necessary permissions.
   • Verify that the index ID is correct.
6. Max Retries: If the maximum number of retries is reached without a final status, the code will print "Max retries reached, exiting." This indicates that the index creation process is taking longer than expected.

By successfully checking the status of the search index creation, you can ensure that your search capabilities are ready for querying in the next steps.

‍

Step 8: Query the Search Index to Find Relevant Videos Based on Your Search Criteria

Now that you have created and verified your search index, you can query it to find relevant videos based on specific search criteria. Follow these steps to perform a search:

1. Set Up the API Request: Use the following code snippet to query the search index. Make sure you have the correct index ID from the previous steps.

search_index_url = f"{base_api_url}/index/{index_id}/search/"

form_data = {
    "query": "tennis on grass",  # Replace with your search query
    "limit": "10"  # Optional: Limit the number of results returned
}

response = requests.post(search_index_url, headers=headers, data=form_data)

results = response.json().get("result_rows")

print("Videos from most to least relevant:\n")
for idx, result in enumerate(results):
    print(f"{idx + 1}: {result[1]}")

2. Customize Your Query: Replace "tennis on grass" in the form_data dictionary with your desired search query. You can also adjust the limit parameter to control how many results you want to retrieve.
3. Execute the Code: Run the code snippet in your Python environment.
4. Check the Response: The code will send a request to the search index and retrieve the results. If the request is successful, it will print the videos ordered from most to least relevant based on your search criteria.
5. Results Interpretation: The results will be displayed in a numbered list format, where each entry corresponds to a video that matches your query. The result part assumes that the second element in the result tuple contains the relevant video information (like the title or URL). Adjust the index if your results structure differs.
6. Troubleshooting: If you encounter issues with the search query:
   • Ensure that your API token is valid and has the necessary permissions.
   • Verify that the index ID is correct and that the index has been successfully created.

By completing this step, you will have effectively queried your search index and retrieved relevant videos based on your specified criteria, showcasing the powerful capabilities of the Twelve Labs Embed API and Roe AI integration.

‍

Conclusion

This tutorial has demonstrated the integration of Twelve Labs' video embedding capabilities with Roe AI's data management platform, providing developers with powerful tools for video search and analysis. By following the steps outlined, you've created a semantic video search solution that showcases the potential of this technology stack.

For developers, this integration opens up new possibilities in video processing. The combination of Twelve Labs' advanced video understanding and Roe AI's flexible data handling enables the creation of more intelligent, context-aware applications. These tools allow developers to process and understand video content in ways that were previously challenging or impractical.

As you continue your development work, consider how you might apply these technologies to your own projects. Whether you're working on content recommendation systems, video analytics tools, or applications dealing with large-scale video data, this integration provides a solid foundation for building sophisticated AI-powered video-native solutions.

‍

Appendix

For your reference and further exploration:

1. Complete Colab Notebook
2. Twelve Labs API documentation
3. Roe AI documentation

We'd love to see what you build! Share your projects and experiences with the Twelve Labs and Roe AI communities. Happy coding!

‍