Tutorial

12 Min

Building a Security Analysis Application with Twelve Labs

Author

Hrishikesh Yadav

Date Published

January 8, 2025

Introduction

Ever wondered how many critical security events go unnoticed in hours of security footage? 🎥🔍 Imagine being able to instantly pinpoint the exact moments that matter, transforming endless video streams into actionable intelligence.

The Security Analysis Application processes security footage, dash camera videos, and CCTV recordings to identify crucial moments within seconds. Built with the Twelve Labs SDK, this powerful application automatically identifies and timestamps key security events—from unauthorized access attempts to suspicious behavior patterns. Security personnel can now skip hours of manual review and instead quickly search for specific incidents using simple prompts, receiving instant results with precise timestamps and confidence scores.

The system enhances detection by generating comprehensive, AI-driven reports for each identified event. Using the Pegasus 1.1 Engine (Generative Engine), it converts raw video data into detailed incident documentation with contextual analysis for actionable insights. This end-to-end solution streamlines security operations and ensures every critical security event is captured, serving both personal and professional security needs. Let's explore how this application works and how you can build similar solutions using the TwelveLabs Python SDK.

You can explore the demo of the application here: Security Analysis

If you want to access the code and experiment with the app directly, you can use this Replit Template.

‍

Demo Application

Below is a demo of the Security Analysis Application, which indexes video footage of car accidents, fights, and paranormal activities. Try out the Security Analysis App.

When you search for "Car Accident on the highway," the API retrieves all relevant videos with confidence scores and timestamps. You can select "detailed analysis" to view an in-depth report generated from the video based on the system prompt. The report can be downloaded as a PDF, enhancing the security system's functionality.

In another demo example, searching for "Paranormal activity identified" retrieves footage of dark and unusual activities, showcasing the application's capabilities.

‍

Prerequisites

Generate an API key by signing up at the Twelve Labs Playground.
Find the repository for this application on Twelve Labs Security Analysis
There are several things you should already be familiar with - Flask, HTML, CSS, JavaScript.

‍

How the Application Works

This section outlines the application flow for developing the Security Analysis Application powered by Twelve Labs Engine.The application requires indexing CCTV and dash cam video footage, which can be done in two ways. The preparation steps section provides instructions for creating the index through Twelve Labs Playground and uploading videos.

The second method uses cloud-to-cloud transfer through an AWS S3 bucket integrated with Twelve Labs for indexing and storing videos. This method is ideal for handling large amounts of video data. For detailed instructions, refer to the Cloud-to-Cloud Integration Guide.

The application architecture consists of three layers: Backend, Frontend, and Service Layer. When you query the Twelve Labs Search API, it returns relevant videos with their video_id, score, confidence, timestamp (start, end, duration), and video_url for rendering.

When you select a video from the search results, the Generate API activates the Pegasus 1.1 Engine (Generative Engine). Using an open-ended prompt, the system analyzes the video to identify and describe any incidents or unusual events. You can then generate and save the analysis as a PDF report.

‍

Preparation Steps

Obtain your API Key from the Twelve Labs Playground and prepare the environment variable.
Clone the project from Github or use the Replit Template
Create and populate a video index, which is essential for search functionality. Add videos through Twelve Labs Playground by creating an Index with the Pegasus 1.1 Engine, then upload your video clips.
Alternatively, you can use the Twelve Labs Integration to load all the footage videos into an AWS S3 bucket. Make sure you follow the instructions in the Twelve Labs - Cloud to Cloud Integration guide.
Set up the .env file with your API Key, along with the main file.

API_KEY=your_api_key_here
INDEX_ID=your_index_id

With these steps completed, you're now ready to dive in and develop the application!

‍

Walkthrough for Security Analysis App

In this tutorial, we'll build a Flask application with a minimal frontend. Here's the directory structure:

.               
├── README.md                 
├── .gitignore           
├── .env                    
├── static/            
│   └── css/
│       └── style.css 
├── templates/     
│   └── index.html
├── app.py 
├── report_generator.py  
├── requirements.txt 
└── uploads/

‍

Creating the Flask Application

Now that you've completed the previous steps, it's time to build the Flask application.

You can find the required dependencies to prepare and create a virtual environment here: requirements.txt

Create a virtual Python environment, then set up the environment for the application using the following command:

pip install -r requirements.txt

‍

1 - Setting up the Main Application

This section focuses on the main application utility function, which contains the core logic and workflow. We'll break down the main application into two key sections:

Setting up the video search engine
Generating detailed analysis reports from searched videos

‍

1.1 Setting up the video search engine

The process begins with importing and initializing the Twelve Labs client to set up the video search API. Our security video analysis platform features robust error handling and efficient search capabilities, supported by a comprehensive error management system that ensures reliable operation and appropriate error reporting.

# To handle errors across routes
def handle_errors(f):
    @wraps(f)
    def decorated_function(*args, **kwargs):
        try:
            return f(*args, **kwargs)
        except Exception as e:
	     # Log Error
            app.logger.error(f"Error: {str(e)}")
            return jsonify({
                'error': 'An error occurred processing your request',
                'details': str(e)
            }), 500
    return decorated_function


@app.route('/search', methods=['POST'])
@handle_errors
def search():
    # Validate request contains JSON data
    if not request.is_json:
        return jsonify({'error': 'Request must be JSON'}), 400
    
    data = request.get_json()
    query = data.get('query')
    
    if not query:
        return jsonify({'error': 'Query parameter is required'}), 400

    try:
       # Search videos using Twelve Labs API
        search_results = client.search.query(
            index_id=INDEX_ID,
            query_text=query,
            options=["visual"]
        )
        
        formatted_results = []
        # Process each video clip from search results
        for clip in search_results.data:
            try:
                headers = {
                    "x-api-key": API_KEY,
                    "Content-Type": "application/json"
                }
                print(headers)
                print(f"{BASE_URL}/indexes/{INDEX_ID}/videos/{clip.video_id}")
               # To Fetch additional video details
                url_response = requests.get(
                    f"{BASE_URL}/indexes/{INDEX_ID}/videos/{clip.video_id}",
                    headers=headers
                )
                
                # Extract video metadata
                video_data = url_response.json()
                video_url = video_data.get('hls', {}).get('video_url')
                video_duration = video_data.get('metadata', {}).get('duration', 0)
                
                formatted_results.append({
                    'video_id': clip.video_id,
                    'score': clip.score,  
                    'confidence': 'High' if clip.score > 0.7 else 'Medium',
                    'start': clip.start,
                    'end': clip.end,
                    'duration': video_duration,
                    'video_url': video_url
                })
            except Exception as e:
                app.logger.error(f"Error getting video URL: {str(e)}")
                continue
        
        return jsonify(formatted_results)
        
    except Exception as e:
        app.logger.error(f"Search Error: {str(e)}")
        return jsonify({'error': 'Search failed', 'details': str(e)}), 500


# Defining the directory path for sample videos
SAMPLE_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'sample')


@app.route('/sample/<path:filename>')
def serve_sample_video(filename):
    # Serve sample video files from the sample directory
    try:
        return send_from_directory(SAMPLE_DIR, filename)
    except Exception as e:
        app.logger.error(f"Error serving video: {str(e)}")
        return f"Error: {str(e)}", 404

Each search request goes through multiple validation steps. First, the system checks the JSON structure and required parameters, providing clear feedback if any validation fails. After validation, the system uses Twelve Labs' search capabilities to find and carefully analyze relevant video clips.

The system processes results using a layered approach for enhanced reliability. For each video clip found, it retrieves additional metadata including URLs and durations, and implements confidence scoring to indicate result relevance. Using a threshold of 0.7, the confidence calculation offers a straightforward way to communicate result quality. If individual results fail to process, the system continues handling the remaining results rather than failing completely.

‍

1.2. Generating detailed analysis reports from searched videos

After retrieving videos from the index based on the provided query, the system generates a detailed analysis of the selected video. The process starts by fetching the video URL through an HTTP GET request to the Twelve Labs API using the specific video ID.

# To handle video analysis requests
@app.route('/analyze/<video_id>')
@handle_errors
def analyze(video_id):
    try:
        app.logger.info(f"Starting analysis for video: {video_id}")
        
        try:
            # 1. Get video URL
            headers = {
                "x-api-key": API_KEY,
                "Content-Type": "application/json"
            }
            url_response = requests.get(
                f"{BASE_URL}/indexes/{INDEX_ID}/videos/{video_id}",
                headers=headers
            )
            url_data = url_response.json()
            video_url = url_data.get('hls', {}).get('video_url')
            
            # 2. Generate analysis
            analysis_response = client.generate.text(
                video_id=video_id,
                prompt="Provide a detailed analysis of the key events, actions, and notable elements in this video."
            )
            
            # Extract and clean analysis text
            analysis_text = str(analysis_response.data) if hasattr(analysis_response, 'data') else ''
            analysis_text = analysis_text.strip().strip('"\'')
            
            response_data = {
                'video_url': video_url,
                'analysis': analysis_text or "No analysis available."
            }
            
            app.logger.info("Analysis completed successfully")
            return jsonify(response_data)
            
        except Exception as e:
            app.logger.error(f"Error in analysis: {str(e)}")
            raise
            
    except Exception as e:
        app.logger.error(f"Analysis Error: {str(e)}")
        return jsonify({
            'error': 'Analysis failed',
            'details': str(e)
        }), 500

Once the streaming URL is obtained, the Twelve Labs client generates text analysis using an open-ended prompt designed for comprehensive video content analysis. The system then cleans and formats the analysis text by removing excess quotes and whitespace.

The final response combines the video URL and analysis text, includes fallback options for missing analyses, and handles logging.

‍

2 - Building an Interactive Analysis Model and Handling Video Player Integration

This section explores the index.html file. The code orchestrates search functionality and real-time video analysis to create a seamless user experience. The application handles DOM interactions, state management, HLS video streaming, and comprehensive error handling. We'll break down the major functionality into these subsections:

Setting up the Analysis Pipeline
HLS Video Player Integration

‍

2.1 Setting up the Analysis Pipeline

Here we'll explore how we implement video analysis and handle errors and UI feedback in our application.

    window.analyzeVideo = async function(videoId) {
        const modalContent = document.getElementById('analysisContent');
        const downloadButton = document.querySelector('.download-button');
        const modal = document.getElementById('analysisModal');

        // Show modal and display loading spinner
        modal.classList.remove('hidden');
        modalContent.innerHTML = `
            <div class="spinner">
                <div class="spinner-ring"></div>
                <p>Analyzing video...</p>
            </div>
        `;
        downloadButton.disabled = true;
    
        try {
            // Log analysis request and fetch results from server
            console.log('Analyzing video:', videoId);
            const response = await fetch(`/analyze/${videoId}`);
            const data = await response.json();
            console.log('Analysis response:', data);
    
            if (!response.ok) {
                console.error('Response not OK:', response.status, response.statusText);
                throw new Error(data.error || 'Analysis failed');
            }
            // Validate analysis text exists
            const analysisText = data.analysis;
            if (!analysisText) {
                console.error('No analysis text available');
                throw new Error('No analysis data available');
            }
          
                // Clean up text by removing escape characters and quotes
   let analysisText = data.analysis
                     .replace(/\\n/g, '\n')
                     .replace(/^"|"$/g, '')
                     .trim();

}

The analyzeVideo function manages the video analysis process. It begins by getting references to the modal, content area, and download button. During analysis, the modal appears and the download button is disabled. The function makes a backend request using the videoId and converts the response to JSON. Then it displays the analysis text after a final validation check.

The final step processes the text, converting raw analysis into well-structured HTML content. This includes converting newlines to line breaks, removing quotes, and trimming whitespace.

‍

2.2 HLS Video Player Integration

This section covers video streaming management using the HLS (HTTP Live Streaming) protocol.

if (data.video_url) { //Check if the video URL exists in response
    const video = document.getElementById('analysis-video');

    // Verify video element exists and browser supports HLS playback
    if (video && Hls.isSupported()) {
        // Initialize and configure HLS player instance
        const hls = new Hls();
        hls.loadSource(data.video_url);
        hls.attachMedia(video);
        
       // Error Handling for video playback
        hls.on(Hls.Events.ERROR, function(event, data) {
            console.error('Video playback error');
            if (data.fatal) {
                handleVideoError();
            }
        });
    }
}
// Store current analysis text in state and enable download
state.currentAnalysis = analysisText;
downloadButton.disabled = false;

} catch (error) {
// Log and display of Error
console.error('Analysis error');
modalContent.innerHTML = `
    <div class="error-message">
        <i class="fas fa-exclamation-circle"></i>
        <p>${error.message || 'Failed to analyze video. Please try again.'}</p>
        <button class="retry-btn" onclick="analyzeVideo('${videoId}')">
            <i class="fas fa-redo"></i> Try Again
        </button>
    </div>
`;
}
};

We start by verifying three key elements: the video URL exists, the video element is present in the DOM, and the browser supports HLS streaming. To set up the HLS player, we create a new instance, load the video source URL, and attach the player to the video element. Once setup is complete, we update the application state with the analysis text and enable downloads. We've added error monitoring to track HLS streaming issues in the console. When fatal errors occur, the handleVideoError function takes over.

You can find the complete Script code in this index.html file.

‍

Conclusion

This blog post explains how to develop and use the security application with Twelve Labs for both professional and personal security needs. Thank you for following this tutorial. We welcome your ideas on improving the user experience and solving various challenges.

‍

Additional Resources

Learn more about Pegasus 1.1 (Generator Engine), the engine used for generation tasks. To further explore Twelve Labs and enhance your understanding of video content analysis, check out these valuable resources:

Discord Community: Join our vibrant community of developers and enthusiasts to discuss ideas, ask questions, and share your projects.
Sample Applications: Explore a variety of sample applications to inspire your next project or learn new implementation techniques.
Explore Tutorials: Dive deeper into Twelve Labs capabilities with our comprehensive tutorials.

We encourage you to leverage these resources to expand your knowledge and create innovative applications using Twelve Labs video understanding technology.

Generation Examples

No items found.

Comparison against existing models