DataFuseAI

Context

DataFuseAI is a multi-tenant, no-code ETL and data-operations platform built by Azminds Services in Kathmandu. Teams connect sources, build pipelines without writing them, run them on a schedule, and ship the whole thing to their own servers if they need to. I joined in September 2024, when it was mostly an idea and a few Django models, and took it to production.

My role

Three roles, one product. Associate software engineer for the first four months: early REST APIs, Celery and Redis background tasks, JWT and OTP auth, the foundations of the tenant model, a connection-profile module, a virtual file explorer with permission scopes, and the first AWS setup. Software engineer from January 2025: the job module, cross-engine queries, the database migration, credential encryption, compiled builds, deployments. Tech lead and product engineer from September 2025: technical and product direction to release, the first AI feature, licensing, the multi-tenancy rewrite, API ingestion, cloud cost, benchmarking, and the docs site.

Constraints

Small team, on-premise customers who couldn’t be sent source code, six database engines that all had to behave the same in a query editor, and an AWS bill that was too large for the stage we were at. I arrived without Django, Docker, or AWS.

What I built

Enlarge:

• Job orchestration. Async execution, cron scheduling on Celery Beat, DAG dependencies between pipeline steps, conditional routing, and cancellation by job or run ID.
• Cross-engine query execution with cancellation and WebSocket streaming across PostgreSQL, MySQL, SQL Server, Oracle, Snowflake, and MongoDB.
• Multi-tenancy, re-architected. An invite → accept → verify flow; one user, many tenants.
• API-source ingestion across backend, frontend, and Spark: 6 auth methods including both OAuth flows, 6 pagination strategies, schema inference, and an SSRF-safe client, tested against a dozen live APIs.
• The first AI feature. A natural-language-to-SQL assistant that introspects schema and never touches user data; I proposed the bring-your-own-key LLM architecture.
• Licensing, end to end — it has its own page.
• Security basics done properly. AES-CBC/PBKDF2 credential encryption and secret masking.
• Shipping to customers’ servers. The Django backend compiled to C binaries with Nuitka; deployments on CentOS, Fedora, Amazon Linux, and Windows; NGINX with Let’s Encrypt; GitHub Actions deploying to EC2; Sentry and structured logging.
• The docs site, solo, on Docusaurus: 50+ technical articles on ETL architecture, change data capture, and data governance, many on Google’s first page, backed by Power BI and Tableau reference implementations.
• The commercial side. Marketing CMS backend, solution pages, the PoC request flow, and the product SWOT and pitch playbook.

Decisions and why

• SQLite → Dockerised PostgreSQL once concurrent writes from parallel job runs became the bottleneck. Diagnosing the contention was the work; the migration was the easy part.
• Nuitka over obfuscation for on-premise shipping: a compiled binary was the only option that met “no source on the customer’s box” without a runtime dependency story. Cost: long build times and four operating systems to keep green.
• Turn things off instead of downsizing. Right-sizing plus automated start and stop on Lambda, EventBridge, and Step Functions cut AWS from $773 to $175 a month — 77% — in two months, because the peak load was real for a few hours a day and idle the rest.
• Benchmark before tuning. TPC-H benchmarking and Spark tuning got pipelines to 60M+ rows at a peak of 64,439 rows per second on a 2-core, 16 GB setup.
• Bring-your-own-key for the AI feature, so customer data and customer LLM spend stayed with the customer.

Outcome

A production release; a documentation site that ranks; and a licensing system and an ingestion layer that other engineers could extend.

What I’d change

Introduce benchmarking earlier, so the Spark tuning starts from measurements. Write the ingestion connector’s auth layer once, as a library, before the fifth auth method rather than after.

Links

datafuseai.com · docs.datafuseai.com — the docs site is my most complete solo artifact from these two years.