API Connector

Context

DataFuseAI could read databases, cloud storage and uploaded files, but not APIs. A lot of public data only comes through an API: weather services, government data portals, financial feeds and research archives. I was asked to research how API sources should work and to build a proof of concept, separate from the main platform. This repository is that proof of concept.

My role

All of it: the research, the design, the Django backend, the React frontend, the tests and the documentation.

Constraints

The server sends requests to any URL a user types, and every response that comes back is untrusted. Each API splits its data into pages in its own way, and stopping after the first page loses data without any error. No two APIs return the same shape, and some return XML instead of JSON. Credentials can never be stored or written to a log in plain text.

What I built

Enlarge:

• Connection profiles and endpoints. A profile holds the base URL, the authentication and the default headers. Each endpoint under it is one path with its method, parameters and pagination, the way a table sits under a database.
• 6 authentication types: no auth, API key, Basic, Bearer token, and OAuth 2.0 with client credentials or with an authorization code and PKCE.
• 6 pagination strategies: limit and offset, page and size, cursor, next URL, the Link header, and none. Every strategy stops at 100 pages or 10,000 records by default, whatever the API sends back.
• Schema inference. It reads every record of a response, turns nested fields into dot paths such as customer.address.city, counts the empty values and picks a type for each field. The user can then include, rename or override each one.
• A connection test in 6 steps: DNS, network, auth injection, HTTP response, format detection and a response sample, so a failure names the step that broke.
• A live data preview of the records a pipeline would receive, with the raw response beside it and export to CSV or JSON.
• XML as well as JSON. Both become one record shape at a single point, so pagination, schema inference and the preview did not change when XML support arrived.
• Security. Credentials are encrypted at rest with Fernet and are never returned or logged. SSRF protection blocks private, loopback and link-local addresses, and XML is parsed with external entities turned off.

Decisions and why

• The pagination engine is a Python generator. It hands over one page at a time, so memory stays at the size of one page, and a preview stops early without fetching pages it does not show.
• Credentials in one encrypted JSON field, not a column for each value, so a new auth type adds no database column.
• One place for encryption. Only the encryption service calls Fernet, and CI fails any change that imports it anywhere else.
• A synchronous HTTP client, httpx, for the first version. One person uses it at a time, and async would have changed every view and every test for no gain yet.
• The 4 simple auth types first, for the platform. Most public APIs need no auth, a key, Basic or a Bearer token. Both OAuth flows work here, but each provider sets them up differently, so I recommended leaving them out of the platform’s first release.

Outcome

A working proof of concept, tested against public APIs such as NASA, GitHub, Hacker News, BBC News and the USGS earthquake feed. Its design maps onto DataFuseAI: a profile becomes a source in the pipeline editor, and each endpoint becomes a table. The API-source ingestion I built for DataFuseAI has the same 6 authentication methods, 6 pagination strategies, schema inference and SSRF protection.

What I’d change

Time the slow paths against real APIs. The benchmark timed the database operations, but the connection test, schema inference and the preview depend on a live API, and none of them was timed against one. I’d also add structured logging before anyone else runs it.

Links

The repository, api-connector, is linked at the top of this page.