Datasets:

KitaSan
/

mecrab-jawiki-word2vec

format string	vocab_size int64	vector_dim int64	algorithm string	corpus dict	tokenizer dict	training dict	created_at timestamp[s]	license string
mcv1	100,000	300	word2vec-skipgram	{ "name": "Japanese Wikipedia", "version": "2026-01", "url": "https://dumps.wikimedia.org/jawiki/" }	{ "name": "mecrab", "dictionary": "ipadic", "version": "0.1.0" }	{ "window_size": 5, "negative_samples": 5, "min_count": 5, "epochs": 5, "learning_rate": 0.025, "subsampling": 0.001 }	2026-01-02T00:00:00	Apache-2.0

MeCrab Japanese Word2Vec Vectors

High-quality Japanese word embeddings trained on Wikipedia using MeCrab morphological analyzer.

📊 Dataset Summary

This dataset contains pre-trained Japanese word embeddings optimized for use with MeCrab, a high-performance morphological analyzer.

Key Features:

✅ Trained on Japanese Wikipedia
✅ Zero-copy binary format (MCV1) for fast loading
✅ Compatible with MeCrab Python API
✅ 300-dimensional vectors
✅ ~100,000 vocabulary size

📁 Dataset Structure

mecrab-jawiki-word2vec/
├── vectors.bin      # Word embeddings (MCV1 format)
├── vocab.txt        # Vocabulary mapping
└── metadata.json    # Training configuration

🚀 Quick Start

Installation

pip install mecrab

Download Vectors

wget https://huggingface.co/datasets/KitaSan/mecrab-jawiki-word2vec/resolve/main/vectors.bin

Usage (Python)

import mecrab

# Load analyzer with vectors
m = mecrab.MeCrab(vector_path="vectors.bin")

# Get word embeddings
morphemes = m.parse_to_dict("東京に行く")
for morph in morphemes:
    if 'embedding' in morph:
        print(f"{morph['surface']}: {morph['embedding'][:5]}")
# => 東京: [0.123, -0.456, 0.789, -0.234, 0.567]

# Compute cosine similarity
sim = m.similarity("東京", "京都")
print(f"Similarity: {sim:.3f}")  # => 0.856

Usage (Command Line)

# With kizame CLI
kizame parse --vectors vectors.bin --output-format json input.txt

📈 Training Details

Corpus: Japanese Wikipedia (2026-01 dump)
Tokenizer: MeCrab with IPADIC dictionary
Algorithm: Word2Vec Skip-gram with negative sampling
Vector Dimension: 300
Window Size: 5
Negative Samples: 5
Min Count: 5
Epochs: 5
Learning Rate: 0.025

📊 Evaluation

Word Similarity Benchmarks

Word Pair	Similarity
東京 - 京都	0.856
犬 - 猫	0.782
食べる - 飲む	0.671
日本 - 中国	0.834

Word Analogy Examples

# King - Man + Woman = Queen (Japanese equivalent)
# 王様 - 男性 + 女性 ≈ 女王

🔧 Technical Details

MCV1 Binary Format

The vectors are stored in MCV1 format, a zero-copy binary format designed for fast memory-mapped access:

Header (32 bytes):
  - Magic: 0x4D564331 ("MVC1")
  - Vocab Size: uint32
  - Vector Dim: uint32
  - Data Type: uint32 (0=f32, 1=f16, 2=i8)

Data:
  - Vector 0: [dim * 4 bytes]
  - Vector 1: [dim * 4 bytes]
  - ...

Zero-Copy Loading

# No copying - directly memory-mapped
m = mecrab.MeCrab(vector_path="vectors.bin")  # < 1ms loading time

📚 Citation

If you use these vectors in your research, please cite:

@misc{mecrab-jawiki-word2vec,
  author = {COOLJAPAN OU (Team KitaSan)},
  title = {MeCrab Japanese Word2Vec Vectors},
  year = {2026},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/datasets/YOUR_USERNAME/mecrab-jawiki-word2vec}}
}