tomaarsen/clap-htsat-fused-librispeech

CLAP model trained on COCO Captions

This is a sentence-transformers model finetuned from laion/clap-htsat-fused on the librispeech_asr dataset. It maps sentences & paragraphs to a None-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: laion/clap-htsat-fused
• Maximum Sequence Length: None tokens
• Output Dimensionality: None dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • librispeech_asr
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'get_text_features', 'method_output_name': None}, 'audio': {'method': 'get_audio_features', 'method_output_name': None}}, 'module_output_name': 'sentence_embedding', 'architecture': 'ClapModel'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/clap-htsat-fused-librispeech")
# Run inference
sentences = [
    'THERE ARE NATURES TOO TO WHOSE SENSE OF JUSTICE THE PRICE EXACTED LOOMS UP MONSTROUSLY ENORMOUS ODIOUS OPPRESSIVE WORRYING HUMILIATING EXTORTIONATE INTOLERABLE THOSE ARE THE FANATICS',
    'HE BEGAN TO WISH THAT HE HAD COMPROMISED IN SOME WAY OR OTHER THAT HE HAD SENT THE MONEY PERHAPS HE COULD DO IT UP HERE',
    'HERE THE HOLY PRELATE OF FERNS MET HIM AND RELATED A VISION IN WHICH HE HAD BEEN INSTRUCTED TO DEMAND THE ABOLITION OF THE IMPOST',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000, -0.4742, -0.2719],
#         [-0.4742,  1.0000,  0.8206],
#         [-0.2719,  0.8206,  1.0000]])

Evaluation

Metrics

Information Retrieval

• Datasets: librispeech-eval and librispeech-test
• Evaluated with InformationRetrievalEvaluator

Metric	librispeech-eval	librispeech-test
cosine_accuracy@1	0.108	0.151
cosine_accuracy@3	0.196	0.288
cosine_accuracy@5	0.272	0.371
cosine_accuracy@10	0.438	0.518
cosine_precision@1	0.108	0.151
cosine_precision@3	0.0653	0.096
cosine_precision@5	0.0544	0.0742
cosine_precision@10	0.0438	0.0518
cosine_recall@1	0.108	0.151
cosine_recall@3	0.196	0.288
cosine_recall@5	0.272	0.371
cosine_recall@10	0.438	0.518
cosine_ndcg@10	0.2432	0.3132
cosine_mrr@10	0.1849	0.2505
cosine_map@100	0.206	0.2694

Training Details

Training Dataset

librispeech_asr

• Dataset: librispeech_asr at 71cacbf
• Size: 132,553 training samples
• Columns: audio and text
• Approximate statistics based on the first 1000 samples:

  	audio	text
  type	dict	string
  details		min: 20 characters mean: 189.15 characters max: 294 characters

• Samples:

  audio	text
  {'path': '374-180298-0000.flac', 'array': array([ 6.92203816e-04, 8.04404495e-04, 8.03834875e-04, ..., -3.02505396e-05, -6.59527450e-06, 1.11444592e-06]), 'sampling_rate': 48000}	CHAPTER SIXTEEN I MIGHT HAVE TOLD YOU OF THE BEGINNING OF THIS LIAISON IN A FEW LINES BUT I WANTED YOU TO SEE EVERY STEP BY WHICH WE CAME I TO AGREE TO WHATEVER MARGUERITE WISHED
  {'path': '374-180298-0001.flac', 'array': array([-9.33515839e-05, -1.25754057e-04, -1.44482241e-04, ..., -2.66165182e-04, -2.03228556e-04, -1.03404833e-04]), 'sampling_rate': 48000}	MARGUERITE TO BE UNABLE TO LIVE APART FROM ME IT WAS THE DAY AFTER THE EVENING WHEN SHE CAME TO SEE ME THAT I SENT HER MANON LESCAUT FROM THAT TIME SEEING THAT I COULD NOT CHANGE MY MISTRESS'S LIFE I CHANGED MY OWN
  {'path': '374-180298-0002.flac', 'array': array([-2.47883319e-04, -2.91854434e-04, -2.82971043e-04, ..., -1.43931946e-04, -1.17829914e-04, -6.32331648e-05]), 'sampling_rate': 48000}	I WISHED ABOVE ALL NOT TO LEAVE MYSELF TIME TO THINK OVER THE POSITION I HAD ACCEPTED FOR IN SPITE OF MYSELF IT WAS A GREAT DISTRESS TO ME THUS MY LIFE GENERALLY SO CALM

• Loss: MultipleNegativesSymmetricRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

librispeech_asr

• Dataset: librispeech_asr at 71cacbf
• Size: 1,000 evaluation samples
• Columns: audio and text
• Approximate statistics based on the first 1000 samples:

  	audio	text
  type	dict	string
  details		min: 8 characters mean: 104.62 characters max: 516 characters

• Samples:

  audio	text
  {'path': '2277-149896-0000.flac', 'array': array([ 0.00179741, 0.00170625, 0.00120927, ..., -0.00144462, -0.00102732, -0.00048062]), 'sampling_rate': 48000}	HE WAS IN A FEVERED STATE OF MIND OWING TO THE BLIGHT HIS WIFE'S ACTION THREATENED TO CAST UPON HIS ENTIRE FUTURE
  {'path': '2277-149896-0001.flac', 'array': array([ 0.00111104, 0.00081758, 0.00021103, ..., -0.00138193, -0.0009173 , -0.00041702]), 'sampling_rate': 48000}	HE WOULD HAVE TO PAY HER THE MONEY WHICH SHE WOULD NOW REGULARLY DEMAND OR THERE WOULD BE TROUBLE IT DID NOT MATTER WHAT HE DID
  {'path': '2277-149896-0002.flac', 'array': array([0.00080266, 0.00088462, 0.00083408, ..., 0.00105488, 0.00083673, 0.00043296]), 'sampling_rate': 48000}	HURSTWOOD WALKED THE FLOOR MENTALLY ARRANGING THE CHIEF POINTS OF HIS SITUATION

• Loss: MultipleNegativesSymmetricRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: True
• fp16: False
• half_precision_backend: None
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss	librispeech-eval_cosine_ndcg@10	librispeech-test_cosine_ndcg@10
-1	-1	-	-	0.0114	-
0.0100	83	3.5908	-	-	-
0.0200	166	2.5371	-	-	-
0.0301	249	2.1799	-	-	-
0.0401	332	2.0415	-	-	-
0.0501	415	1.9394	-	-	-
0.0601	498	1.8167	-	-	-
0.0701	581	1.7589	-	-	-
0.0801	664	1.7262	-	-	-
0.0902	747	1.7585	-	-	-
0.1001	829	-	1.5991	0.0335	-
0.1002	830	1.7521	-	-	-
0.1102	913	1.6822	-	-	-
0.1202	996	1.6176	-	-	-
0.1302	1079	1.6391	-	-	-
0.1403	1162	1.6931	-	-	-
0.1503	1245	1.4626	-	-	-
0.1603	1328	1.4305	-	-	-
0.1703	1411	1.4998	-	-	-
0.1803	1494	1.4073	-	-	-
0.1903	1577	1.3843	-	-	-
0.2001	1658	-	1.2227	0.0925	-
0.2004	1660	1.3371	-	-	-
0.2104	1743	1.3908	-	-	-
0.2204	1826	1.2835	-	-	-
0.2304	1909	1.3203	-	-	-
0.2404	1992	1.2549	-	-	-
0.2505	2075	1.2384	-	-	-
0.2605	2158	1.2189	-	-	-
0.2705	2241	1.1658	-	-	-
0.2805	2324	1.1771	-	-	-
0.2905	2407	1.2068	-	-	-
0.3002	2487	-	1.0471	0.1318	-
0.3005	2490	1.1708	-	-	-
0.3106	2573	1.1389	-	-	-
0.3206	2656	1.0786	-	-	-
0.3306	2739	1.0792	-	-	-
0.3406	2822	1.0562	-	-	-
0.3506	2905	0.98	-	-	-
0.3607	2988	1.1153	-	-	-
0.3707	3071	0.9987	-	-	-
0.3807	3154	1.0002	-	-	-
0.3907	3237	1.0017	-	-	-
0.4002	3316	-	0.8901	0.1589	-
0.4007	3320	0.9364	-	-	-
0.4107	3403	0.9394	-	-	-
0.4208	3486	0.9459	-	-	-
0.4308	3569	0.9604	-	-	-
0.4408	3652	0.9491	-	-	-
0.4508	3735	0.9295	-	-	-
0.4608	3818	0.9508	-	-	-
0.4709	3901	0.9122	-	-	-
0.4809	3984	0.8483	-	-	-
0.4909	4067	0.8443	-	-	-
0.5003	4145	-	0.7955	0.1908	-
0.5009	4150	0.8838	-	-	-
0.5109	4233	0.8367	-	-	-
0.5209	4316	0.8516	-	-	-
0.5310	4399	0.8112	-	-	-
0.5410	4482	0.8368	-	-	-
0.5510	4565	0.873	-	-	-
0.5610	4648	0.8156	-	-	-
0.5710	4731	0.8864	-	-	-
0.5811	4814	0.8278	-	-	-
0.5911	4897	0.8006	-	-	-
0.6004	4974	-	0.7649	0.1874	-
0.6011	4980	0.8199	-	-	-
0.6111	5063	0.7475	-	-	-
0.6211	5146	0.7345	-	-	-
0.6311	5229	0.7301	-	-	-
0.6412	5312	0.774	-	-	-
0.6512	5395	0.7391	-	-	-
0.6612	5478	0.6929	-	-	-
0.6712	5561	0.7218	-	-	-
0.6812	5644	0.7071	-	-	-
0.6912	5727	0.7024	-	-	-
0.7004	5803	-	0.6712	0.2419	-
0.7013	5810	0.6428	-	-	-
0.7113	5893	0.6719	-	-	-
0.7213	5976	0.6972	-	-	-
0.7313	6059	0.7043	-	-	-
0.7413	6142	0.663	-	-	-
0.7514	6225	0.6963	-	-	-
0.7614	6308	0.6591	-	-	-
0.7714	6391	0.6736	-	-	-
0.7814	6474	0.7033	-	-	-
0.7914	6557	0.6314	-	-	-
0.8005	6632	-	0.6806	0.2319	-
0.8014	6640	0.6508	-	-	-
0.8115	6723	0.6532	-	-	-
0.8215	6806	0.6788	-	-	-
0.8315	6889	0.6038	-	-	-
0.8415	6972	0.658	-	-	-
0.8515	7055	0.656	-	-	-
0.8616	7138	0.6533	-	-	-
0.8716	7221	0.601	-	-	-
0.8816	7304	0.6243	-	-	-
0.8916	7387	0.6315	-	-	-
0.9005	7461	-	0.6526	0.2432	-
0.9016	7470	0.5707	-	-	-
0.9116	7553	0.5778	-	-	-
0.9217	7636	0.5736	-	-	-
0.9317	7719	0.615	-	-	-
0.9417	7802	0.5756	-	-	-
0.9517	7885	0.5724	-	-	-
0.9617	7968	0.5678	-	-	-
0.9718	8051	0.5661	-	-	-
0.9818	8134	0.6162	-	-	-
0.9918	8217	0.5766	-	-	-
-1	-1	-	-	-	0.3132

Environmental Impact

Carbon emissions were measured using CodeCarbon.

• Energy Consumed: 0.429 kWh
• Carbon Emitted: 0.115 kg of CO2
• Hours Used: 2.094 hours

Training Hardware

• On Cloud: No
• GPU Model: 1 x NVIDIA GeForce RTX 3090
• CPU Model: 13th Gen Intel(R) Core(TM) i7-13700K
• RAM Size: 31.78 GB

Framework Versions

• Python: 3.11.6
• Sentence Transformers: 5.2.0.dev0
• Transformers: 4.57.0.dev0
• PyTorch: 2.8.0+cu128
• Accelerate: 1.6.0
• Datasets: 3.6.0
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}