DDLG
Distributed Data Lake Gateway
A high-performance Electron & Rust-based desktop platform enabling distributed data indexing, semantic retrieval, offline AI document classification (using Qwen2.5-0.5B), and custom JavaScript/Python data transformer pipelines across 100M+ corporate files without centralizing data.
01. Vision
DDLG is a cloud-free data management solution that helps teams automate workflows, securely share files, and simplify repetitive tasks.
KEY FEATURES
- ▹ Distributed Indexing (100M+ files)
- ▹ Local AI (Qwen2.5-0.5B offline)
- ▹ Semantic Vector Search (all-mpnet-base-v2)
- ▹ JavaScript/Python Data Transformers
- ▹ Whisper.cpp Meeting Assistant
- ▹ Electron & Rust core integration
02. Project Story & Deep-Dive
The Data Silo Nightmare
Corporate teams were wasting hours trying to find files scattered across employees' laptops. Centralizing all these files in the cloud was impossible due to massive storage costs, network bandwidth limits, and strict data privacy regulations. We needed a system that could index and search files where they live, locally, without a single central file repository.
Architectural Focus
[Data Distribution Status] Total Devices: 550+ Indexed Files: 100M+ Sync Latency: Real-time local search Network Overheads: 0 MB (No Cloud)
Distributed Search & Local AI
We designed a lightweight indexing daemon that runs on each device. It uses Rust for high-speed metadata extraction and Sentence-Transformers ('all-mpnet-base-v2') locally to index and search content semantically. To enable document classification offline, we bundled a quantized Qwen2.5-0.5B model running directly on client devices, eliminating external API dependencies and subscription costs.
Architectural Focus
// Local Vector Inference import { pipeline } from '@xenova/transformers'; const embedder = await pipeline('feature-extraction', 'Xenova/all-mpnet-base-v2' ); const vector = await embedder('search query', { pooling: 'mean', normalize: true });
Optimizing Vector Embeddings on the Edge
Generating and searching high-dimensional vector embeddings on standard office laptops presents severe memory issues. By implementing localized quantization of embeddings and building a highly optimized index pruning algorithm, we cut local database storage by 78% and reduced search queries from days to under 50 milliseconds.
Architectural Focus
// Qwen2.5 GGUF Local Execution const runner = new ElectronLLMRunner({ model: "qwen2.5-0.5b-instruct-gguf", contextLimit: 2048, threads: 4, gpuFallback: false });
