Notes and Comments

[finleyh]

• I would consider naming your web_connectors as such for naming consistency. Not a huge deal but for readability i have a good idea of where i am when im navigating the directories of the project.
• Broader design question: You aren't. going for stateful connections on this connector library. Cool ,but are there situations or usecases where statefulness you get with a class will be missed? Like are there situations where you may want a Session and not just a one off request? Depends on the API I guess?
  *. If I use your library in Airflow, or in Temporal, so think high-throughput microservice making many calls, batch jobs making 1000s of lookups, does this introduce a problem? (IDK) Think bulk or batch lookups

Every function call repeats environment variable parsing and validation. I dont think this is going to kill you, but it is a design feature based on this build out.

Running in the same theme, the creation (for example in Twilio) of the HTTPBasicAuth object happens with every call of the function. Again, it's cheap, but if I'm implementing connectors in a microservice does this have an impact? (IDK)

• imlications with running many function calls in a loop?

I wonder if there is a class based design we could discuss or work through that could help solve some of what you are attempting to do with Broker while also consolidating some of what you are doingin your functions in your API/DBMS specificc modules.

I think this project is a great start. I'm happy to spend some days next week ideating on how we can continue to build on this excellent foundation you have laid out.

[finleyh]

I had a ChatGPT mock up a baseclass that URLScan and Twilio (for example) could inherit from that take care of some of the things you may want to only do once. It leverages your broker to define the 'make_requests' function:

from ppp_connectors.connectors.broker import make_request

class BaseClient:
def init(self, required_env_vars: List[str], auth_builder=None):
self.env_config = self.load_and_validate_env(required_env_vars)
self.auth = auth_builder(self.env_config) if auth_builder else None

@staticmethod
def load_and_validate_env(required_vars):
    config = combine_env_configs()
    check_required_env_vars(config, required_vars)
    return config

def make_request(
    self,
    method,
    url,
    params=None,
    data=None,
    json=None,
    extra_headers=None,
    timeout=None
):
    headers = self.default_headers.copy()
    if extra_headers:
        headers.update(extra_headers)

    return make_request(
        method=method,
        url=url,
        headers=headers,
        auth=self.auth,
        params=params,
        data=data,
        json=json,
        timeout=timeout or 15
    )

[finleyh]

and here is an example of URLScanClient using BaseClient to make request:

class URLScanClient(BaseClient):
BASE_URL = "https://urlscan.io/api/v1"

def __init__(self):
    super().__init__(required_env_vars=['URLSCAN_API_KEY'])
    self.api_key = self.env_config['URLSCAN_API_KEY']
    self.default_headers = {
        'accept': 'application/json',
        'API-Key': self.api_key
    }

def search(self, query, **kwargs):
    return self.make_request(
        'get',
        f"{self.BASE_URL}/search/",
        params={'q': query, **kwargs}
    )

def scan(self, url_to_scan, **kwargs):
    payload = {'url': url_to_scan, **kwargs}
    return self.make_request(
        'post',
        f"{self.BASE_URL}/scan",
        json=payload
    )

[finleyh]

And here are all the reasons why your broker design is smart, and good:

✅ 1️⃣ Broker separates transport from business logic

Your broker.make_request() function is pure HTTP:
✔️ Method → GET, POST, etc.
✔️ URL → Any
✔️ Headers
✔️ Params / JSON / Data
✔️ Timeout
✔️ Proxies
✔️ SSL verify

✅ It doesn't know or care what service you're talking to.

That is good.

---

Key design principle: Separation of Concerns.

✅ Broker handles transport-level details.
✅ Clients handle service-level details.

---

✅ 2️⃣ Broker centralizes shared HTTP configuration

✔️ Reads global proxy settings once:

PPP_HTTP_PROXY
PPP_HTTPS_PROXY

✔️ Reads SSL verification flag:

VERIFY_SSL

✔️ Handles disabling warnings.

✅ If you want to change proxy behavior for every service, you change one place.

Without broker?
❌ You’d duplicate this code in every client.
❌ Error-prone and unmaintainable.

---

✅ 3️⃣ Broker gives you consistent request behavior

✅ You have one place to:

• Define proxy settings
• Handle SSL verification
• Set timeouts

✅ Any change (e.g. adding logging, retry logic) improves every service instantly.

---

✅ 4️⃣ Broker enables easy cross-cutting features

Because it’s centralized, you can easily add:

✔️ Global logging

logging.info(f"{method} {url} called")

✔️ Retry logic

with backoff:
    request_func(...)

✔️ Metrics

increment_metric("api_call", service=service_name)

Without broker?
You'd have to duplicate that logic everywhere.

---

✅ 5️⃣ Broker is perfect for testing/mocking

✅ If you want to test clients without hitting real APIs:

✔️ You can mock broker.make_request in tests:

@patch("broker.make_request")
def test_client_lookup(mock_request):
    ...

✅ No need to patch every service class separately.

---

✅ 6️⃣ Broker ensures environment consistency

✔️ It guarantees all requests obey your env-based settings:

• Proxies
• SSL verify

✅ You can deploy in different environments with just env vars.

---

✅ 7️⃣ Broker is dead simple

✅ It’s not a class, session, or complicated state machine.
✅ Just one function anyone can read in 30 seconds.
✅ Trivially testable.

---

✅ 8️⃣ Broker is easy to replace

✅ If tomorrow you wanted to use httpx instead of requests?

You replace just broker.make_request.

All your service clients keep working without changes.

---

✅ 9️⃣ Broker matches real SDK designs

✔️ boto3
→ Has low-level HTTP transport layer.
→ High-level service clients just call transport.

✔️ google-cloud-python
→ Has a shared HttpTransport.
→ Service classes call it.

✔️ Stripe/Plaid/Sendgrid SDKs
→ Usually have one transport layer class or function.

✅ Broker is that transport layer.

---

✅ 10️⃣ TL;DR – Why do I like broker?

Because it’s clean separation of concerns.

✔️ Broker = Universal, environment-aware, transport layer
✔️ BaseClient = Adds service-specific config (headers, auth)
✔️ ServiceClient = Adds business logic (endpoints, parameters)

✅ Keeps your design modular, DRY, testable, and maintainable.

---

✅ Without broker?
❌ Each client has to manage its own proxy/env/SSL/timeouts.
❌ Duplication everywhere.
❌ Higher chance of bugs.

✅ With broker?
✔️ One transport function to rule them all.
✔️ Clients just focus on what they need to do.

---

✅ Bottom line

🟢 Broker is a great foundation for building clean, maintainable API client code.

[robd518]

💯 ❤

Plan moving forward, I will:

For DBMS connectors only

• drop the broker pattern all together so that we can use the most powerful functionality of each DBMS without forcing each DBMS to adhere to a fundamentally flawed broker pattern
• move DBMS connections to be both class-based and stateful
• consider and attempt to implement easy-to-use functions for less-technical users that don't want to have to manage classes and state

For API connectors only, after the above is complete

• remove environment variable checking from each connector; when I initially made this pattern it was with the intent of aligning ppp_connectors to "the framework" primarily; this is no longer the case and it doesn't make sense to do it this way.
• IMPORTANT NOTE - I need to either patch in a shim to still use environment variables or time the release of these updates with a mass-update everywhere that uses ppp_connectors (this isn't impossible, it just requires thought).
• reconsider functional vs class-based; needs consideration and discussion, just need to finish other items first.

For all

• redo all test cases
• redo README

[robd518]

Addressing in https://github.com/FineYoungCannibals/ppp_connectors/tree/dbms-connectors