So I’m no expert, but I have been a hobbyist C and Rust dev for a while now, and I’ve installed tons of programs from GitHub and whatnot that required manual compilation or other hoops to jump through, but I am constantly befuddled installing python apps. They seem to always need a very specific (often outdated) version of python, require a bunch of venv nonsense, googling gives tons of outdated info that no longer works, and generally seem incredibly not portable. As someone who doesn’t work in python, it seems more obtuse than any other language’s ecosystem. Why is it like this?
The reason you do stuff in a venv is to isolate that environment from other python projects on your system, so one Python project doesn’t break another. I use Docker for similar reasons for a lot of non-Python projects.
A lot of Python projects involve specific versions of libraries, because things break. I’ve had similar issues with non-Python projects. I’m not sure I’d say Python is particularly worse about it.
There are tools in place that can make the sharing of Python projects incredibly easy and portable and consistent, but I only ever see the best maintained projects using them unfortunately.
Python is hacky, because it hacks. There’s a bunch of ways you can do anything. You can run it on numerous platforms, or even on web assembly. It’s not maintained centrally. Each “app” you find is just somebodies hack project they’re sharing with you for fun.
Python is the new Perl
Docker might be solution here.
But from my experience most python scripts are absolute junk. The barrier for entry is low so there’s a massive disparity in quality.
everyone focuses on the tooling, not many are focusing on the reason: python is extremely dynamic. like, magic dynamic you can modify a module halfway through an import, you can replace class attributes and automatically propagate to instances, you can decompile the bytecode while it’s running.
combine this with the fact that it’s installed by default and used basically everywhere and you get an environment that needs to be carefully managed for the sake of the system.
js has this packaging system down pat, but it has the advantage that it got mainstream in a sandboxed isolated environment before it started leaking out into the system. python was in there from the beginning, and every change breaks someone’s workflow.
the closest language to look at for packaging is probably lua, which has similar issues. however since lua is usually not a standalone application platform it’s not a big deal there.
It’s something of a “14 competing standards” situation, but uv seems to be the nerd favourite these days.
I still do the python3 -m venv venv && source venv/bin/activate
How can uv help me be a better person?
And pip install -r requirements.txt
Fuck it, I just use sudo and live with the consequences.
This! Haven’t used that one personally, but seeing how good ruff is I bet it’s darn amazing, next best thing that I used has been PDM and Poetry, because Python’s first party tooling has always been lackluster, no cohesive way to define a project and actually work it until relatively recently
I moved all our projects (and devs) from poetry to uv. Reasons were poetry’s non standard pyproject.toml syntax and speed, plus some weird quirks, e. g. if poetry asks for input and is not run with the verbose flag, devs often don’t notice and believe it is stuck (even though it’s in the default project README).
Personally, I update uv on my local machine as soon as a new release is available so I can track any breaking changes. Couple of months in, I can say there were some hiccups in the beginning, but currently, it’s smooth sailing, and the speed gain really affects productivity as well, mostly due to being able to not break away from a mental “flow” state while staring at updates, becoming suspicious something might be wrong. Don’t get me wrong, apart from the custom syntax (poetry partially predates the pyproject PEP), poetry worked great for us for years, but uv feels nicer.
Recently, “uv build” was introduced, which simplified things. I wish there was an command to update the lock file while also updating the dependency specs in the project file. I ran some command today and by accident discovered that custom dependency groups (apart from e. g. “dev”) have made it to uv, too.
“uv pip” does some things differently, in particular when resolving packages (it’s possible to switch to pip’s behavior now), but I do agree with the decisions, in particular the changes to prevent “dependency confusion” attacks.
As for the original question: Python really has a bit of a history of project management and build tools, I do feel however that the community and maintainers are finally getting somewhere.
cargo is a bit of an “unfair” comparison since its development happened much more aligned with Rust and its whole ecosystem and not as an afterthought by third party developers, but I agree: cargo is definitely a great benchmark how project and dependency management plus building should look like, along with rustup, it really makes the developer experience quite pleasant.
The need for virtual environments exists so that different projects can use different versions of dependencies and those dependencies can be installed in a project specific location vs a global, system location. Since Python is interpreted, these dependencies need to stick around for the lifetime of the program so they can be imported at runtime. poetry managed those in a separate folder in e. g. the user’s cache directory, whereas uv for example stores the virtual environment in the project folder, which I strongly prefer.
cargo will download the matching dependencies (along with doing some caching) and link the correct version to the project, so a conceptual virtual environment doesn’t need to exist for Rust. By default, rust links everything apart from the C runtime statically, so the dependencies are no longer neesed after the build - except you probably want to rebuild the project later, so there is some caching.
Finally, I’d also recommend to go and try setting up a project using astral’s uv. It handles sane pyproject.toml files, will create/initialize new projects from a template, manages virtual environments and has CLI to build e. g. wheels or source distribution (you will need to specify which build backend to use. I use hatchling), but thats just a decision you make and express as one line in the project file. Note: hatchling is the build backend, hatch is pypa’s project management, pretty much an alternative to poetry or uv.
uv will also install complete Python distributions (e. g. Python 3.12) if you need a different interpreter version for compatibility reasons
If you use workspaces in cargo, uv also does those.
uv init, uv add, uv lock --upgrade, uv sync, uv build and how uv handles tools you might want to install and run should really go a long way and probably provide an experience somewhat similar to cargo.
Python’s packaging is not great. Pip and venvs help but, it’s lightyears behind anything you’re used to. My go-to is using a venv for everything.
You re not stupid, python’s packaging & versionning is PITA. as long as you write it for yourself, you re good. As soon as you want to share it, you have a problem
as long as you write it for yourself, you re good. As soon as you want to share it, you have a problem
A perfect summary of the history of computer code!
Python never had much of a central design team. People mostly just scratched their own itch, so you get lots of different tools that do only a small part each, and aren’t necessarily compatible.
With all the hype surrounding Python it’s easy to forget that it’s a really old language. And, in my opinion, the leadership is a bit of a mess so there hasn’t been any concerted effort on standardizing tooling.
Some unsolicited advice from somebody who is used more refined build environments but is doing a lot of Python these days:
The whole
venvthing isn’t too bad once you get the hang of it. But be prepared for people to tell you that you’re using the wrong venv for reasons you’ll never quit understand or likely need to care about. Just use the bundled “python -m venv venv” and you’ll be fine despite other “better” alternatives. It’s bundled so it’s always available to you. And feel free to just drop/recreate your venv whenever you like or need. They’re ephemeral and pretty large once you’ve installed a lot of things.Use “pipx” to install python applications you want to use as programs rather than libraries. It creates and manages venvs for them so you don’t get library conflicts. Something like “pip-tools” for example (pipx install pip-tools).
Use “pyenv” to manage installed python versions - it’s a bit like “sdkman” for the JVM ecosystem and makes it easy to deal with the “specific versions of python” stuff.
For dependencies for an app - I just create a requirements.txt and “pip install -r requirements.txt” for the most part… Though I should use one of the 80 better ways to do it because they can help with updating versions automatically. Those tools mostly also just spit out a requirements.txt in the end so it’s pretty easy to migrate to them. pip-tools is what my team is moving towards and it seems a reasonable option. YMMV.
I agree. Python is my language of choice 80% or so of the time.
But my god, it does packaging badly! Especially if it’s dependent on linking to compiled code!
Why it is like that, I couldn’t tell. The language is older than git, so that might be part of it.
However, you’re installing python libraries from github? I very very rarely have to do that. In what context do you have to do that regularly?
venv nonsense
I mean, the fact that it isn’t more end-user invisible to me is annoying, and I wish that it could also include a version of Python, but I think that venv is pretty reasonable. It handles non-systemwide library versioning in what I’d call a reasonably straightforward way. Once you know how to do it, works the same way for each Python program.
Honestly, if there were just a frontend on venv that set up any missing environment and activated the venv, I’d be fine with it.
And I don’t do much Python development, so this isn’t from a “Python awesome” standpoint.
pyenv and uv let you install and switch between multiple Python versions.
As for uv, those come from the Python build standalone project, if I remember correctly, pyenv also installs from there, but don’t quote me on that.
This is exactly how I feel about python as well… IMHO, it’s good for some advanced stuff, where bash starts to hit its limits, but I’d never touch it otherwise
Yeah the tooling sucks. The only tooling I’ve liked is Poetry, I never have trouble installing or packaging the apps that use it.
It… depends. There is some great tooling for Python – this was less true only a few years ago, mind you – but the landscape is very much in flux, and usage of the modern stuff is not yet widespread. And a lot of the legacy stuff has a whole host of pitfalls.
Things are broadly progressing in the right direction, and I’d say I’m cautiously optimistic, although if you have to deal with anything related to conda then for the time being: good luck, and sorry.
Tried to install Automatic1111 for Stable Diffusion in an Arch distrobox, and despite editing the .sh file to point to the older tarballed Python version as advised on Github, it still tells me it uses the most up to date one that’s installed system wide and thus can’t install pytorch. And that’s pretty much where my personal knowledge ends, and apparently that of those (i.e. that one person) on Github. ¯\_(ツ)_/¯
Always funny when people urge you to ask for help but no one ends up actually helping.
despite editing the .sh file to point to the older tarballed Python version as advised on Github, it still tells me it uses the most up to date one that’s installed system wide and thus can’t install pytorch.
Can you paste your commands and output?
If you want, maybe on !imageai@sh.itjust.works, since I think that people seeing how to get Automatic1111 set up might help others.
I’ve set it up myself, and I don’t mind taking a stab at getting it working, especially if it might help get others over the hump to a local Automatic1111 installation.
