Speeding Up API Development

Most of my projects these days involve building a web API. Building a web API using REST principles allows you to bolt on multiple different front-ends (e.g., React app or iOS/Android app) and easily access functions from other computers. But building the infrastructure of the API often takes time, time I’d really like to spend building out cool functions instead. So over time I’ve learnt a number of tricks to speed up the basics.

FastAPI

Like many folks, I tend to start with a Python environment with FastAPI as my base library.

I like to separate different layers of the API:

• Routes / Endpoints
• CRUD Methods
• Data Models

CRUD stands for Create-Read-Update-Delete and represents different operations that may be performed on the data.

In parallel, I can then define functions that work on the data models. These can be triggered by calls to particular endpoints or by internal processing:

• Logic / Processing Functions
• Data Models

Folder Structure

I tend to use a folder structure that reflects the above layers:

• api
  • crud – a folder with CRUD methods for each data model
  • routes – a folder with different files that define different routes in a modular fashion
  • schemas – a folder with different files that hold different Pydantic schemas
  • main.py – a short file that initialises the FastAPI object and sets the configuration
  • dependencies.py – a file for any dependencies that are used across multiple files, avoids circular imports
• config – folder that with files that setup the general configuration
• database – folder with files that setup the database
• logic – folder with files that handle general processing
• models – folder with files that define the data models
• tests – folder with tests that are developed as we go along

General Process

Here is my general process for coding up a web API:

• Define data model(s)
• Write tests for data models
• Define Pydantic models corresponding to data models
• Define CRUD methods that receive and return Pydantic models and modify the data models
• Write tests for the CRUD methods
• Define routes that correspond to the CRUD methods
• Write tests for the routes
• Write advanced logic
• Write tests for advanced logic
• Define routes for initialising advanced logic
• Write tests for logic routes

Data Models

When developing I tend to start with the data models.

The data models are defined as a set of classes. If you are using SQLAlchemy, these may inherit from a Base class. Properties are defined as part of each class definition. When using a database ORM like SQLAlchemy or Tortoise, the classes get mapped to database tables and the properties to fields in each table.

I try to define data models in a modular manner. It’s useful to use a rough one-one mapping between files and models (e.g., my User model class is in a file called users.py).

Pydantic Models

While the data models define what is stored in the database, the Pydantic models define what data is sent across the external-facing API.

There is often a lot of repetition but (folks say) it is good practice to separate the internal and external models for greater control. I typically start by copying the data model definitions and then editing those to restrict certain fields.

CRUDRouter

One trick to simplify the CRUD methods and routes is to use the CRUDRouter library.

Using this library you can add API routes based on the Pydantic models.

The CRUDRouter library has router objects that can take three Pydantic schemas:

• a general schema that is the main Pydantic schema
• a create schema that is used for creating new objects – generally this omits the id
• an update schema that is used for updating existing objects – by limiting the fields on this object you can limit the updateable fields

The SQLAlchemy example can be found here. You can leave the create_schema argument blank on initialisation and the primary key id will be stripped automatically from the general schema to create the create schema. Similarly, if you leave out the update_schema argument, an update schema will be automatically created using all the fields of the general schema.

The code implementation of the SQLAlchemyCRUDRouter is straightforward and can be found here. It’s probably worth keeping your implementation simple to use the CRUDRouter as a short cut. It also makes testing straightforward as you can use parameterised testing to test multiple models and routes.

If you do use the CRUDRouter, routes for the advanced logic can be added to the defined router object in a simple manner. I would still use one file per set of object routes to keep things modular and expandable. In this manner you can do away with the crud folder and CRUD tests.

Using the CRUDRouter, you can skip three steps of development and simplify route testing. In this manner you can concentrate on just the data models and the advanced logic.

Automation

This whole process is fairly mechanical. This leads me to think that you could code up a GPT/LLM application that automated the procedure. This is on the to-do list.