r/Python • u/Daraminix • 1d ago
Discussion Running 60 python scripts as "services"
Hi,
I have around 60 scripts that need to run constantly, mainly event handlers and such. Right now I have an external script that launch them and monitor if the app is running on it's pid, otherwise it's relaunching the app. Works fine but get's clunky when we update some submodule and need to restart them, or to check if one crashes more than other etc..
So I would like to find a better way to approach this. It needs to run on windows and being able to access several samba shares via unc paths and being able to restart crashed scripts anf offer an easy way to restart all of them in case of an update (this part doesn't need to be automated). Every script use the same environnement
For now my candidates are docker, PM2 and NSSM.
I think docker is gonna be a pain to access shares and add a lot of overhead especially on windows
I don't know PM2 and NSSM, looks like PM2 would be easier to setup but more JS oriented and NSSM would be harder to monitor.
What do you think guys ?
25
u/ivanational 1d ago
The urge to say "just switch to Linux" is strong here lol. But seriously, avoid Docker for this on Windows. Give PM2 a shot - it handles Python fine and pm2 logs will quickly show you which of those 60 scripts is throwing a tantrum
3
u/mumrik1 8h ago
Why would you avoid Docker for this?
4
u/ivanational 8h ago
Mainly because of the overhead and unnecessary complexity for a pure process-management task. Running docker desktop on Windows means dealing with WSL2/Hyper-v resource consumption, and if those 60 scripts need access to local network shares (SMB), file mounting in Windows Docker becomes an absolute pain.
24
u/jonathon8903 1d ago
Are you married to running it on Windows? If not it would be semi-trivial to setup those shared folders on Linux as mounted directories and then just pass them in as docker volumes. Then deployment becomes extremely easy.
1
u/Daraminix 1d ago
Not married to windows but our whole codebase was targeted for windows, would need to add some path substitutions and don't want to deal with file permissions issues and such fun things :)
Would be easier to keep the same environement as everything else3
u/jonathon8903 1d ago
Okay so yeah long term it may be beneficial to migrate to environment variables and containerize to make it easier for deployment. But if you need a short term win then I agree with another suggestion here to just use NSSM. I used it about 5 years ago and it was pretty solid. There might be other, better options I'm not aware of though. I haven't touched Windows in about two years now.
8
u/Aggressive_You6518 1d ago
pm2 works fine with python, just set the interpreter in the config and you're good, it's not that js-locked as people think
docker on windows is indeed asking for pain especially with unc paths, you'll spend more time fighting permissions than actually running your scripts
1
10
u/james_d_rustles 20h ago
This is besides the point, but I just want to say that this is a high quality thread. This particular problem is outside of my scope so Iām just an observer, but itās refreshing to see a well defined problem/question and a thread full of knowledgeable humans sharing their 2 cents and experiences.
Itās very banal and unimportant in the grand scheme of things, but lately a lot of coding subs seem to be nothing but āshould I learn {language}???ā or ācheck out this slop repo Gemini made me, please give me GitHub stars and congratulate meā, or some variation of those. Iāve always had a hunch that people actually learn a lot from casually perusing stackoverflow, forums and whatnot, just in the sense that we pick up all sorts of little nuggets of information that seem inconsequential but add up to a deeper understanding of the field over time, and since AI got big I feel like I see less and less of those types of conversations.
Itās nice to occasionally be reminded that there are still knowledgeable humans out there, is all.
Sorry for the ramble, donāt mind me - just sharing some thoughts.
7
9
u/JimroidZeus 1d ago
I agree that docker would be too much overhead, will cause problems accessing shares if not setup properly.
PM2 seems like itās node/bun specific, so Iād suggest trying out NSSM.
For what itās worth, I do use docker to run Python apps on Windows and itās not too terrible to get setup. I donāt have anything accessing samba shares though.
4
u/greenearrow 1d ago
Iāve been using nssm for years - when they were only a couple years past their last update. I fear the day the lack of support bites me in the ass though. Their website has a tendency to be inaccessible at times.
2
3
u/el_extrano 1d ago
I've used NSSM to daemonize python on Windows, and logging to disk for observability. It works but feels a little hacky, mainly because with NSSM, the nssm.exe executable is what will show as the executable for each service. Plus, I don't like that NSSM is just a binary you download from the internet, it's not open source. Not to mention, it makes packaging a problem, because any target computer needs nssm in order to install the services, but iirc you are not supposed to distribute copies of the binary yourself.
I'd question whether you really need 60 services? Could you compose it into 1 entrypoint actually managed as a Windows service, with the rest of the microservices managed inside Python using async, threading, or multiprocessing?
3
u/ivanational 1d ago
Spot on. Managing 60 individual processes instead of refactoring into async/multiprocessing in a single core app is just treating the symptoms, not the cause
2
u/Daraminix 1d ago
Yeah we thought about refactoring it, some of them are actually running with a single entrypoint and launched as separate threads. The issue is that we are using an api developped by a thrid-party provider and the session management throught the server doesn't do well when threaded or asynced (cache corruption for example). It's an ongoing issue we have with them but they are more inclined to put up to speed their js api than fixing issues in their python one.
Furthermore some scripts launch complex software in cli mode that can crash for multiple reasons and we cannot just relaunch everyting when of of the process crashes. Bundling everything into one entrypoint means one bad crash can take down unrelated scripts unless we build very careful supervision/restart logic around each unit of work. At that point we are just reimplementing what the OS process model and a service manager already gives us for free1
u/HommeMusical 12h ago
Sounds miserable! Your solution is, well, ugly but practical and simple, and reduces risk. Sounds very real-world and useful.
The issue is that we are using an api developped by a thrid-party provider
We're in the days of vibe coding.
My friend got a new job and replaced a SaaS service that the company was paying $640k a year for by a mostly vibe-coded app he wrote in a couple of months.
They're running the two in parallel now, and the vibe coded application seems more reliable than the paid one.
Just a thought!
NOTE NOTE NOTE: My friend has been programming for decades, almost as long as I have, and he's a no-bullshit programmer who is very skeptical. You wouldn't get such good results with a junior programmer.
2
u/Daraminix 10h ago
Yeah we could probably rewrite or make some changes in the api but too much things revolve around it to risk it. And I don't want to add another thing to maintain and make update of our central tool harder
3
u/roxalu 1d ago
NSSM was a great tool - but nowadays there exist actively maintained better alternatives as e.g. https://github.com/aelassas/servy
3
u/pacopac25 1d ago
Use Servy to install it as a service.
https://github.com/aelassas/servy
I'd install one main script that runs the others.
You can also use Task Scheduler in Windows, but if you want them "installed" as services, Servy is great.
3
u/FunkyMonkey237 21h ago edited 13h ago
Are these 60 scripts all running under a different process? The efficacy lover in me is shuddering at the thought of all that wasted CPU and RAM. Python is already inefficient but multiply that by 60 and yikes!
My first thought, without seeing your setup, would be to run them all under the same python process. I'd have a custom manger designed to load, execute and monitor them. Ideally asynchronous but if that's too much effort to rewrite the existing code and there is a risk of a single script blocking, then I'd look at threading based approach (not multi processor!). I haven't tested it but maybe the new GIL free interpreter could be used if compute bound.
Also, are the scripts really 60 unique scripts? Are there overlaps in functionality, could you collapse certain ones into a more generalised script?
4
u/its_a_gibibyte 1d ago
Tell us more about the event handlers. Can you migrate all 60 scripts into a web framework? And then have each one as different http endpoints? And then one event handler/dispatcher to capture local events if you need it. Would make testing and deployment much cleaner as well.
4
u/Zizizizz 1d ago
As overkill as it may be, what popped into my head was containerising each script, maybe look at the Google https://github.com/GoogleContainerTools/distroless which help you build small images.
Publish each to an image repo when you version them.
And reference them as pods in a kubernetes deployment (helm is nice and AI smashes those yaml files). If running locally you can probably use k3s (iirc).Ā
Then you set up and install something like https://fluxcd.io/ in the cluster to reach out to GitHub and pull new images / versions of the scripts you reach.
So each script would be testable and standalone, configured via k8s config and environment variables.
And automatically refresh/update when new commits are pushed (with the ability to rollback easily), and will automatically retry if the image crashes.
Might be overkill but these setups are easier to setup today than they used to be.Ā
3
1
1
u/Individual-Flow9158 1d ago edited 1d ago
Is there an IaC way (e.g. Ansible for Windows targets), or a replicateable way to create installers, and so set up each script as a native persistent Windows Service? Say what you like about Windows, the OS is excellent at relaunching processes (try as you might to kill some of them, they just won't stay dead).
I'd ask on a dedicated Windows or IT sub. Windows Sysadmins have secret powers (thankfully that don't all rely on powershell).
1
u/Scrapheaper 1d ago
What's on the shares? How often is it updated? Why do you have to use fileshares?
1
u/bernasIST 1d ago
Good thing that no one mentioned Windows Task Scheduler... In my org we use it to manage 350 python scripts and it works fine but it is everything on prem and CI/CD breaks. So we will give it a shot with dockers hosted in aws and then use airflow... We are not sure yet how we will manage this migration and if this setup will work as expected... I wonder where I could get advice on this as well.. following this thread
1
1
u/mortenb123 8h ago
On windows I've been using https://nssm.cc/ everywhere for anything 'running a service as a user deployed to a server or workstation' for years from winNT to win11. its just a single binary you attach to your script deployer with a few lines of code to install it.
0
u/chief_kennoh 14h ago
You could run them using supervisor but I have a better solution. I have recently developed platform that has provisions to upload your python scripts, configure an appropriate execution schedule, adjust any variables/parameters (if any) dynamically and even upload associated files. The platform will take care of execution and logging you with options to alert you in the event of execution completion. We offer 1 month free trial with 250 execution minutes for your use. You can check it out on the link below.
43
u/dent308 1d ago edited 1d ago
supervisord may be a good fit,
edit: nevermind, windows