Back-end6 minute read

Streamline Your Django Settings With Type Hints: A Pydantic Tutorial, Part 1

Python and pydantic make a powerful runtime combination when validating data and settings, resulting in reduced debugging and a simplified code base.


Toptalauthors are vetted experts in their fields and write on topics in which they have demonstrated experience. All of our content is peer reviewed and validated by Toptal experts in the same field.

Python and pydantic make a powerful runtime combination when validating data and settings, resulting in reduced debugging and a simplified code base.


Toptalauthors are vetted experts in their fields and write on topics in which they have demonstrated experience. All of our content is peer reviewed and validated by Toptal experts in the same field.
Arjaan Buijk
Verified Expert in Engineering

Arjaan is a senior engineer and data scientist who creates mission-critical cloud solutions focused on Rasa for international banks and insurance companies. He architects and teaches large-scale Kubernetes solutions.

Expertise

PREVIOUSLY AT

Rasa
Share

This is the first installment in a series on leveraging pydantic for Django-based projects. Part 2 builds an app on this concept with Docker and conda to show how to align development and production environments; part 3 explores how to deploy a Django app to Heroku.

Django projects used to frustrate me because I lacked a robust and scalable way to add new environments. By bringing together pydantic and Python type hints, I built the powerful foundation I needed.

As described in PEP 484, type hints support static analysis, but these same annotations are also available at runtime. Third-party packages like pydantic offer runtime type checking that uses this additional metadata. Pydantic uses Python type hints to help manage settings metadata and perform runtime data validation.

This pydantic tutorial will show the far-reaching, positive effects of using pydantic settings management with Django.

Our configuration adheres to the best practices described on the Twelve-Factor App website:

  1. Define nonconstant and secret configurations as environment variables.
  2. In development environments, define environment variables in a .env file and add the .env to .gitignore.
  3. Use the cloud provider’s mechanisms to define (secret) environment variables for the QA, staging, and production environments.
  4. Use a single settings.py file that configures itself from the environment variables.
  5. Use pydantic to read, check, validate, and typecast environment variables onto Python variables that define the Django configurations.

Alternatively, some developers create multiple settings files, like settings_dev.py and settings_prod.py. Unfortunately, this approach does not scale well. It leads to code duplication, confusion, hard-to-find bugs, and higher maintenance efforts.

Using the aforementioned best practices, adding any number of environments is easy, well-defined, and error-proof. Although we could explore a more complicated environment configuration, we will focus on two for clarity: development and production.

What does this look like in practice?

Pydantic Settings Management and Environment Variables

We now focus on an example in both development and production. We show how each environment configures its settings differently and how pydantic supports each.

Our example application requires a Django-supported database, so we need to store the database connection string. We move the database connection configuration information into an environment variable, DATABASE_URL, using the Python package dj-database-url. Please note that this variable is of type str and is formatted as follows:

postgres://{user}:{password}@{hostname}:{port}/{database-name}
mysql://{user}:{password}@{hostname}:{port}/{database-name}
oracle://{user}:{password}@{hostname}:{port}/{database-name}
sqlite:///PATH

In our development environment, we can use a Docker-contained PostgreSQL instance for ease of use, while in our production environment, we will point to a provisioned database service.

Another variable we want to define is a boolean, DEBUG. The DEBUG flag in Django must never be turned on in a production deployment. It is intended to get additional feedback during development. For example, in debug mode, Django will display detailed error pages when an exception occurs.

Different values for development and production could be defined as follows:

Variable NameDevelopmentProduction
DATABASE_URLpostgres://postgres:mypw@localhost:5432/mydbpostgres://foo1:foo2@foo3:5432/foo4
DEBUGTrueFalse

We use the pydantic settings management module to manage these different sets of environment variable values depending on the environment.

Preparatory Steps

To put this into practice, we start configuring our development environment by creating our single .env file with this content:

DATABASE_URL=postgres://postgres:mypw@localhost:5432/mydb
DEBUG=True

Next, we add the .env file to the project’s .gitignore file. The .gitignore file avoids saving potentially sensitive information in source control.

Whereas this approach works well in our development environment, our production environment specification uses a different mechanism. Our best practices dictate that production environment variables use environment secrets. For example, on Heroku, these secrets are called Config Vars and are configured through the Heroku Dashboard. They are made available to the deployed application as environment variables:

A screenshot of the Config Vars web interface. The left sidebar has a description: "Config vars change the way your app behaves. In addition to creating your own, some add-ons come with their own." The main section has two rows, each with two text fields, a pencil icon, and an X icon. The text fields have the same data as the "Variable Name" and "Production" columns in the previous table. The upper-right corner has a button that says, "Hide Config Vars."

After that, we need to adjust the application’s configuration to read these values from either environment automatically.

Configuring Django’s settings.py

Let’s start with a new Django project to provide the essential structure for our example. We scaffold a new Django project with the following terminal command:

$ django-admin startproject mysite

We now have a basic project structure for the mysite project. The project’s file structure is as follows:

mysite/
    manage.py
    mysite/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py

The settings.py file contains boilerplate code that allows us to manage the application configuration. It has many predefined default settings that we must adjust to the relevant environment.

To manage these application settings using environment variables and pydantic, add this code to the top of the settings.py file:

import os
from pathlib import Path
from pydantic import (
    BaseSettings,
    PostgresDsn,
    EmailStr,
    HttpUrl,
)
import dj_database_url

# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent

class SettingsFromEnvironment(BaseSettings):
    """Defines environment variables with their types and optional defaults"""

    DATABASE_URL: PostgresDsn
    DEBUG: bool = False

    class Config:
        """Defines configuration for pydantic environment loading"""

        env_file = str(BASE_DIR / ".env")
        case_sensitive = True

config = SettingsFromEnvironment()

os.environ["DATABASE_URL"] = config.DATABASE_URL
DATABASES = {
    "default": dj_database_url.config(conn_max_age=600, ssl_require=True)
}
DEBUG = config.DEBUG

This code does the following:

  • Defines a class SettingsFromEnvironment, inheriting from pydantic’s BaseSettings class.
  • Defines DATABASE_URL and DEBUG, setting their type and optional default using Python type hints.
  • Defines a class Config telling pydantic to look for the variables in a .env file if not present in the system’s environment variables.
  • Instantiates the Config class into the object config; the desired variables become available as config.DATABASE_URL and config.DEBUG.
  • Defines the regular Django variables DATABASES and DEBUG from these config members.

The same code runs in all environments, and pydantic takes care of the following:

  • It looks for the environment variables DATABASE_URL and DEBUG.
    • If defined as environment variables, like in production, it will use those.
    • Otherwise, it pulls those values from the .env file.
    • If it doesn’t find a value, it will do the following:
      • For DATABASE_URL, it throws an error.
      • For DEBUG, it assigns a default value of False.
  • If it finds an environment variable, it will check field types and give an error if either of them are wrong:
    • For DATABASE_URL, it verifies that its field type is a PostgresDsn-style URL.
    • For DEBUG, it verifies that its field type is a valid, nonstrict pydantic boolean.

Please note the explicit setting of the operating system’s environment variable from the configuration value for DATABASE_URL. It may seem redundant to set os.environ["DATABASE_URL"] = config.DATABASE_URL because DATABASE_URL is already defined as an external environment variable. However, this allows pydantic to parse, check, and validate this variable. If the environment variable DATABASE_URL is missing or formatted incorrectly, pydantic will give a clear error message. These error checks are invaluable as the application moves from development to subsequent environments.

If a variable is not defined, either a default will be assigned or an error will prompt for it to be defined. Any generated prompts also detail the desired variable type. A side benefit of these checks is that new team members and DevOps engineers more easily discover which variables need to be defined. This avoids the hard-to-find issues that result when the application runs without all the variables defined.

That’s it. The application now has a maintainable, settings-management implementation using a single version of settings.py. The beauty of this approach is that it allows us to specify the correct environment variables in a .env file or any other desired means available through our hosting environment.

The Scalable Path

I have been using Django settings management with pydantic runtime typing in all of my Django projects. I have found it leads to smaller, more maintainable codebases. It also provides a well-structured, self-documenting, and scalable approach to adding new environments.

The next article in this series is a step-by-step tutorial on building a Django application from scratch, with pydantic settings management; we then cover deploying it to Heroku.


The Toptal Engineering Blog extends its gratitude to Stephen Davidson for reviewing the code samples presented in this article.

An earlier version of this article emphasized a specific Python release. Since Python has supported type hints for several years, we have generalized the introductory text by removing this version reference.

Understanding the basics

  • What is pydantic?

    Pydantic is a Python package that enforces type hints at runtime.

  • What is pydantic used for?

    Pydantic provides data validation and settings management.

  • Should you use Python type hints?

    Yes, you should use Python type hints because your code becomes self-documenting. You can enforce static type safety with mypy and runtime type safety with pydantic.

  • Do type hints make Python faster?

    Type hints do not make Python faster or slower, but they will speed up your development process by avoiding bugs and hard-to-find runtime errors.

Hire a Toptal expert on this topic.
Hire Now
Arjaan Buijk

Arjaan Buijk

Verified Expert in Engineering

Plymouth, MI, United States

Member since June 4, 2018

About the author

Arjaan is a senior engineer and data scientist who creates mission-critical cloud solutions focused on Rasa for international banks and insurance companies. He architects and teaches large-scale Kubernetes solutions.

authors are vetted experts in their fields and write on topics in which they have demonstrated experience. All of our content is peer reviewed and validated by Toptal experts in the same field.

Expertise

PREVIOUSLY AT

Rasa

World-class articles, delivered weekly.

By entering your email, you are agreeing to our privacy policy.

World-class articles, delivered weekly.

By entering your email, you are agreeing to our privacy policy.

Join the Toptal® community.