Skip to content

Commit 482b847

Browse files
fede-bellofede-bello
committed
doc: add lazy loading documentation
1 parent 6b688c2 commit 482b847

File tree

1 file changed

+286
-0
lines changed

1 file changed

+286
-0
lines changed

‎docs/index.md‎

Lines changed: 286 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2932,3 +2932,289 @@ mutable_settings.__init__()
29322932
print(mutable_settings.foo)
29332933
#> foo
29342934
```
2935+
2936+
## Lazy Loading
2937+
2938+
Lazy loading defers field value resolution until fields are actually accessed, rather than eagerly fetching all values during settings initialization. This is particularly useful when working with cloud secret managers where each field access triggers an API call, avoiding unnecessary network requests for fields that may never be used.
2939+
2940+
### Overview
2941+
2942+
By default, pydantic-settings eagerly resolves all field values from all configured sources during initialization. For cloud secret managers like AWS Secrets Manager, Azure Key Vault, or Google Cloud Secret Manager, this means every field triggers an API call to fetch the secret value, even if the application never uses that field.
2943+
2944+
**When to use lazy loading:**
2945+
2946+
* You use cloud secret managers (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager)
2947+
* Your settings have many fields but your application only uses a subset of them
2948+
* You want to reduce initialization time and API call costs
2949+
* Network latency to secret managers is significant
2950+
2951+
**Trade-offs:**
2952+
2953+
* Fields are resolved when first accessed, not during initialization, so errors surface later
2954+
* There's a small overhead on first access to each field (caching minimizes subsequent accesses)
2955+
* `model_dump()` and other full-model operations will trigger resolution of all fields
2956+
2957+
### Basic Usage
2958+
2959+
You can enable lazy loading in two ways:
2960+
2961+
**1. Global configuration via SettingsConfigDict:**
2962+
2963+
```py
2964+
from pydantic_settings import BaseSettings, SettingsConfigDict
2965+
2966+
2967+
class Settings(BaseSettings):
2968+
model_config = SettingsConfigDict(lazy_load=True)
2969+
2970+
api_key: str
2971+
database_url: str
2972+
debug_mode: bool
2973+
```
2974+
2975+
**2. Per-source configuration via settings_customise_sources:**
2976+
2977+
```py
2978+
import os
2979+
2980+
from pydantic_settings import (
2981+
AWSSecretsManagerSettingsSource,
2982+
BaseSettings,
2983+
PydanticBaseSettingsSource,
2984+
)
2985+
2986+
2987+
class Settings(BaseSettings):
2988+
api_key: str
2989+
database_url: str
2990+
2991+
@classmethod
2992+
def settings_customise_sources(
2993+
cls,
2994+
settings_cls: type[BaseSettings],
2995+
init_settings: PydanticBaseSettingsSource,
2996+
env_settings: PydanticBaseSettingsSource,
2997+
dotenv_settings: PydanticBaseSettingsSource,
2998+
file_secret_settings: PydanticBaseSettingsSource,
2999+
) -> tuple[PydanticBaseSettingsSource, ...]:
3000+
aws_settings = AWSSecretsManagerSettingsSource(
3001+
settings_cls,
3002+
secret_id=os.environ['AWS_SECRET_ID'],
3003+
lazy_load=True, # Enable lazy loading only for AWS Secrets Manager
3004+
)
3005+
return (
3006+
init_settings,
3007+
env_settings,
3008+
dotenv_settings,
3009+
aws_settings,
3010+
file_secret_settings,
3011+
)
3012+
```
3013+
3014+
### Cloud Secret Managers with Lazy Loading
3015+
3016+
Lazy loading provides significant performance benefits when using cloud secret managers. Here are examples for each provider:
3017+
3018+
#### AWS Secrets Manager
3019+
3020+
```py
3021+
import os
3022+
3023+
from pydantic import Field
3024+
3025+
from pydantic_settings import (
3026+
AWSSecretsManagerSettingsSource,
3027+
BaseSettings,
3028+
PydanticBaseSettingsSource,
3029+
)
3030+
3031+
3032+
class Settings(BaseSettings):
3033+
# These fields will only be fetched from AWS Secrets Manager when accessed
3034+
api_key: str = Field(validation_alias='MyApiKey')
3035+
database_password: str = Field(validation_alias='DbPassword')
3036+
encryption_key: str = Field(validation_alias='EncryptionKey')
3037+
3038+
@classmethod
3039+
def settings_customise_sources(
3040+
cls,
3041+
settings_cls: type[BaseSettings],
3042+
init_settings: PydanticBaseSettingsSource,
3043+
env_settings: PydanticBaseSettingsSource,
3044+
dotenv_settings: PydanticBaseSettingsSource,
3045+
file_secret_settings: PydanticBaseSettingsSource,
3046+
) -> tuple[PydanticBaseSettingsSource, ...]:
3047+
aws_secrets = AWSSecretsManagerSettingsSource(
3048+
settings_cls,
3049+
secret_id=os.environ['AWS_SECRETS_MANAGER_SECRET_ID'],
3050+
lazy_load=True,
3051+
)
3052+
return (
3053+
init_settings,
3054+
env_settings,
3055+
dotenv_settings,
3056+
aws_secrets,
3057+
file_secret_settings,
3058+
)
3059+
3060+
3061+
# Initialization is fast - no API calls made yet
3062+
settings = Settings()
3063+
3064+
# First access to api_key triggers AWS API call
3065+
print(settings.api_key)
3066+
3067+
# Second access uses cached value - no additional API call
3068+
print(settings.api_key)
3069+
```
3070+
3071+
#### Azure Key Vault
3072+
3073+
```py
3074+
from azure.identity import DefaultAzureCredential
3075+
3076+
from pydantic import Field
3077+
3078+
from pydantic_settings import (
3079+
AzureKeyVaultSettingsSource,
3080+
BaseSettings,
3081+
PydanticBaseSettingsSource,
3082+
)
3083+
3084+
3085+
class Settings(BaseSettings):
3086+
api_key: str = Field(validation_alias='MyApiKey')
3087+
database_password: str = Field(validation_alias='DbPassword')
3088+
3089+
@classmethod
3090+
def settings_customise_sources(
3091+
cls,
3092+
settings_cls: type[BaseSettings],
3093+
init_settings: PydanticBaseSettingsSource,
3094+
env_settings: PydanticBaseSettingsSource,
3095+
dotenv_settings: PydanticBaseSettingsSource,
3096+
file_secret_settings: PydanticBaseSettingsSource,
3097+
) -> tuple[PydanticBaseSettingsSource, ...]:
3098+
azure_keyvault = AzureKeyVaultSettingsSource(
3099+
settings_cls,
3100+
vault_url='https://myvault.vault.azure.net/',
3101+
credential=DefaultAzureCredential(),
3102+
lazy_load=True,
3103+
)
3104+
return (
3105+
init_settings,
3106+
env_settings,
3107+
dotenv_settings,
3108+
azure_keyvault,
3109+
file_secret_settings,
3110+
)
3111+
3112+
3113+
# Quick initialization
3114+
settings = Settings()
3115+
3116+
# Fields resolved on first access
3117+
print(settings.api_key)
3118+
```
3119+
3120+
#### Google Cloud Secret Manager
3121+
3122+
```py
3123+
import os
3124+
3125+
from pydantic import Field
3126+
3127+
from pydantic_settings import (
3128+
BaseSettings,
3129+
GoogleSecretManagerSettingsSource,
3130+
PydanticBaseSettingsSource,
3131+
)
3132+
3133+
3134+
class Settings(BaseSettings):
3135+
api_key: str = Field(validation_alias='MyApiKey')
3136+
database_password: str = Field(validation_alias='DbPassword')
3137+
3138+
@classmethod
3139+
def settings_customise_sources(
3140+
cls,
3141+
settings_cls: type[BaseSettings],
3142+
init_settings: PydanticBaseSettingsSource,
3143+
env_settings: PydanticBaseSettingsSource,
3144+
dotenv_settings: PydanticBaseSettingsSource,
3145+
file_secret_settings: PydanticBaseSettingsSource,
3146+
) -> tuple[PydanticBaseSettingsSource, ...]:
3147+
gcp_secrets = GoogleSecretManagerSettingsSource(
3148+
settings_cls,
3149+
project_id=os.environ.get('GCP_PROJECT_ID', 'my-project'),
3150+
lazy_load=True,
3151+
)
3152+
return (
3153+
init_settings,
3154+
env_settings,
3155+
dotenv_settings,
3156+
gcp_secrets,
3157+
file_secret_settings,
3158+
)
3159+
3160+
3161+
# Fast initialization - no GCP API calls
3162+
settings = Settings()
3163+
3164+
# First access triggers GCP Secret Manager API call
3165+
print(settings.api_key)
3166+
```
3167+
3168+
### Behavior and Caching
3169+
3170+
When lazy loading is enabled:
3171+
3172+
1. **Initialization**: Settings are created with minimal overhead. Sources return empty dictionaries instead of eagerly fetching all values.
3173+
3174+
2. **First Access**: When you access a field for the first time (e.g., `settings.api_key`), the value is fetched from the configured source and cached in memory.
3175+
3176+
3. **Subsequent Access**: Accessing the same field again returns the cached value without making another API call.
3177+
3178+
4. **All Fields**: Iteration over all fields (via `model_dump()`, etc.) will trigger resolution of all fields at once.
3179+
3180+
```py
3181+
from pydantic_settings import (
3182+
AWSSecretsManagerSettingsSource,
3183+
BaseSettings,
3184+
PydanticBaseSettingsSource,
3185+
)
3186+
3187+
3188+
class Settings(BaseSettings):
3189+
secret1: str
3190+
secret2: str
3191+
3192+
@classmethod
3193+
def settings_customise_sources(
3194+
cls,
3195+
settings_cls: type[BaseSettings],
3196+
init_settings: PydanticBaseSettingsSource,
3197+
env_settings: PydanticBaseSettingsSource,
3198+
dotenv_settings: PydanticBaseSettingsSource,
3199+
file_secret_settings: PydanticBaseSettingsSource,
3200+
) -> tuple[PydanticBaseSettingsSource, ...]:
3201+
aws_settings = AWSSecretsManagerSettingsSource(
3202+
settings_cls, secret_id='my-secret', lazy_load=True
3203+
)
3204+
return (init_settings, env_settings, dotenv_settings, aws_settings, file_secret_settings)
3205+
3206+
3207+
settings = Settings()
3208+
3209+
# Only secret1 is fetched from AWS Secrets Manager
3210+
print(settings.secret1)
3211+
3212+
# secret1 is already cached, so this doesn't trigger another API call
3213+
print(settings.secret1)
3214+
3215+
# secret2 is fetched on first access
3216+
print(settings.secret2)
3217+
3218+
# This triggers fetching of all remaining fields not yet accessed
3219+
all_values = settings.model_dump()
3220+
```

0 commit comments

Comments
 (0)