rom1504

End goal: have a function keeping only interesting video platform links. Having this would enable getting billions of such links via cc2dataset

This function is a binary classifier. To evaluate it we need links that naturally occur in common crawl. Criteria:

• link not containing video that can be downloaded by yt-dlp should be discarded
• "Bad" links (eg porn) should be discarded in vast majority

To collect this eval set we can:

rom1504

"""
Can you improve it to avoid reading the whole tar file to count the number of samples?
"""

import json
import concurrent.futures

import tarfile
import fsspec
import io

rom1504

"""
This is a deduplication method using pyspark.

input: table with id and 2 columns that contain float values
2 items are considered the same if the float values are equal with a threshold of 0.05

algo: multiply the 2 columns by 1/0.05, resulting in 2 longs. Then use pyspark to perform exact dedup using these 2 columns
Pyspark does distributed sort then linear dedup, so this scales to 100B
"""

rom1504

import wandb

import os
import numpy as np
import time
from os import listdir
import uuid
import sys

path = "/fsx/home-rom1504/"

rom1504

from pyspark.sql import SparkSession
import os
import sys
from pyspark import SparkContext
from pyspark.sql.functions import rand
from pyspark.sql import SparkSession
import random
import math
import time
import boto3

rom1504

See https://gist.github.com/rom1504/67ada3dedbecc113ae2dbdfd9c642d83 for step by step about spark jars

rom1504

import numpy as np
from scipy.fftpack import dct


def hash_algo(pil_img, size=10):
    """
    Get perceptual hash of the input image.

    Args:
        image_array: numpy array that corresponds to the image.

rom1504

Steps:

1. wget https://gist.github.com/rom1504/67ada3dedbecc113ae2dbdfd9c642d83/raw/865fb35e00f21330b5b82aeb7c31941b6c18f649/spark_on_slurm.sh
2. wget https://gist.github.com/rom1504/67ada3dedbecc113ae2dbdfd9c642d83/raw/865fb35e00f21330b5b82aeb7c31941b6c18f649/worker_spark_on_slurm.sh
3. wget https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz && tar xf spark-3.3.1-bin-hadoop3.tgz
4. sbatch spark_on_slurm.sh
5. build a venv, install pyspark, then run something like this:

(you can get https://huggingface.co/datasets/laion/laion-coco/resolve/main/part-00000-2256f782-126f-4dc6-b9c6-e6757637749d-c000.snappy.parquet as an example parquet)

rom1504

#!/bin/bash
#SBATCH --partition=gpu
#SBATCH --job-name=gputest
#SBATCH --nodes 1
#SBATCH --ntasks-per-node 8
#SBATCH --cpus-per-gpu=6
#SBATCH --gres=gpu:8
#SBATCH --nodelist gpu-st-p4d-24xlarge-42 
#SBATCH --output=%x_%j.out
#SBATCH --open-mode=append

rom1504

python3.8 -m venv .env
source .env/bin/activate
pip install -U pip
pip install "jax[cuda11_cudnn82]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html

python
import jax
jax.default_backend()
jax.devices()