> Normally, debuing the compiler is fairly straightforward: it is more or less a run of the mill executable.
> In the bootstrap process, the entire thing becomes way more complex. You see, rustc is not invoked directly. The bootstrap script calls a wrapper around the compiler.
> Running that wrapped rustc is not easy to run either: it requires a whole lot of complex, environment flags to be set.
> All that is to say: I don’t know how to debug the Rust compiler. I am 99.9 % sure there is an easy way to do this, documented somewhere I did not think to look. After I post this, somebody will tell me "oh, you just need to do X".
> Still, at the time of writing, I did not know how to do this.
> So, can we attach gdb to the running process? Nope, it crashes way to quickly for that.
It's kind of funny how often this problem crops up and the variety of tricks I have in my back to deal with it. Sometimes I patch the script to invoke gdb --args [the original command] instead, but this is only really worthwhile if it's a simple shell script and also I can track where stdin/stdout are going. Otherwise I might patch the code to sleep a bit before actually running anything to give me a chance to attach GDB. On some platforms you can get notified of process execs and sometimes even intercept that (e.g. as an EDR solution) and sometimes I will use that to suspend the process before it gets a chance to launch. But I kind of wish there was a better way to do this in general…LLDB has a "wait for launch" flag but it just spins in a loop waiting for new processes and it can't catch anything that dies too early.
I have a C library (I've also done a Python one in the past) that you load into the executable you want to debug. It activated based on an environment variable so normally I just permanently link it.
When it is loaded it will automatically talk to VSCode and tell it to start a debugger and attach to it & it waits for the debugger to attach.
End result is you just have to run your script with an environment variable set and it will automatically attach a nice GUI debugger to the process no matter how deeply buried in scripts and Makefiles it is.
I currently use this for debugging C++ libraries that are dynamically loaded into Questa (a commercial SystemVerilog simulator) that is started by a Python script running in some custom build system.
In the past I used it to debug Python code running in an interpreter launched by a C library loaded by Questa started by a Makefile started by a different Python interpreter that was launched by another Makefile. Yeah. It wasn't the only reason by a long shot but that company did not survive...
Now wondering whether the author might be able to force a core dump. With recent snapshot abilities, on Modern Intel processors one can get a Processor Trace that can be helpful even without getting an actual interactive debugging session (haven't done one of those in a while as snapshots seem enough for my needs these days).
I have a LD_PRELOAD library that hooks SIGSEGV into spawning gdb on the process using the best guess for the process's terminal (which currently isn't very smart because I haven't yet needed to debug processes that do a lot of stdio redirection).
I agree, recently I was working with a large Java program and after spending about 90 minutes (far too long, I was getting obsessed), I just gave up trying to get it into a debugger.
This is one area where rust disappoints me, there isn’t a “cargo debug” built in (there is an external program but it doesn’t work well), and when I just manually attach gdb most of the symbols are usually missing.
I would seriously consider a language billed as “debugger-first”, just to see what the experience was like.
It may not seem like it, but this is impressive progress. Getting a compiler to bootstrap at all is an accomplishment, especially for Rust since that depends on so many things working. Once it can reliably bootstrap, a lot of performance-improving steps can begin. Congrats!
Sounds like attempting to always inline a recursive function should be an error instead. But it's probably undesirable to make that change because it would likely break existing crates and thus backwards compatibility as well?
Someone mentioned recently that the slowness of rustc is in large part due to llvm. I know that is probably orthogonal to the work here, but I do like the idea of building the compiler with different toolchains, and that there may be follow on effects down the line.
Depends on the workload, but yes, codegen is a huge part of the total compilation time.
That said, that doesn't mean LLVM is always where the fixes need to be. For instance, one reason rustc spends a lot of time in LLVM is that rustc feeds more code to LLVM than it should, and relies on the LLVM optimizer to improve it. Over time, we're getting better about how much code we throw at LLVM, and that's providing performance improvements.
It's slow because the borrow checker is NP complete. LLVM may or may not generate slower code than GCC would for rustc, but I doubt it's anywhere close to the primary cause of the lack of snappy.
You're wrong it's been debunked that the borrow checker is any appreciable part of the compile time - Steve Klabnik actually verified it on here somewhere.
> Normally, debuing the compiler is fairly straightforward: it is more or less a run of the mill executable.
> In the bootstrap process, the entire thing becomes way more complex. You see, rustc is not invoked directly. The bootstrap script calls a wrapper around the compiler.
> Running that wrapped rustc is not easy to run either: it requires a whole lot of complex, environment flags to be set.
> All that is to say: I don’t know how to debug the Rust compiler. I am 99.9 % sure there is an easy way to do this, documented somewhere I did not think to look. After I post this, somebody will tell me "oh, you just need to do X".
> Still, at the time of writing, I did not know how to do this.
> So, can we attach gdb to the running process? Nope, it crashes way to quickly for that.
It's kind of funny how often this problem crops up and the variety of tricks I have in my back to deal with it. Sometimes I patch the script to invoke gdb --args [the original command] instead, but this is only really worthwhile if it's a simple shell script and also I can track where stdin/stdout are going. Otherwise I might patch the code to sleep a bit before actually running anything to give me a chance to attach GDB. On some platforms you can get notified of process execs and sometimes even intercept that (e.g. as an EDR solution) and sometimes I will use that to suspend the process before it gets a chance to launch. But I kind of wish there was a better way to do this in general…LLDB has a "wait for launch" flag but it just spins in a loop waiting for new processes and it can't catch anything that dies too early.
I have a C library (I've also done a Python one in the past) that you load into the executable you want to debug. It activated based on an environment variable so normally I just permanently link it.
When it is loaded it will automatically talk to VSCode and tell it to start a debugger and attach to it & it waits for the debugger to attach.
End result is you just have to run your script with an environment variable set and it will automatically attach a nice GUI debugger to the process no matter how deeply buried in scripts and Makefiles it is.
https://github.com/Timmmm/autodebug
I currently use this for debugging C++ libraries that are dynamically loaded into Questa (a commercial SystemVerilog simulator) that is started by a Python script running in some custom build system.
In the past I used it to debug Python code running in an interpreter launched by a C library loaded by Questa started by a Makefile started by a different Python interpreter that was launched by another Makefile. Yeah. It wasn't the only reason by a long shot but that company did not survive...
Now wondering whether the author might be able to force a core dump. With recent snapshot abilities, on Modern Intel processors one can get a Processor Trace that can be helpful even without getting an actual interactive debugging session (haven't done one of those in a while as snapshots seem enough for my needs these days).
I have a LD_PRELOAD library that hooks SIGSEGV into spawning gdb on the process using the best guess for the process's terminal (which currently isn't very smart because I haven't yet needed to debug processes that do a lot of stdio redirection).
I agree, recently I was working with a large Java program and after spending about 90 minutes (far too long, I was getting obsessed), I just gave up trying to get it into a debugger.
This is one area where rust disappoints me, there isn’t a “cargo debug” built in (there is an external program but it doesn’t work well), and when I just manually attach gdb most of the symbols are usually missing.
I would seriously consider a language billed as “debugger-first”, just to see what the experience was like.
> I would seriously consider a language billed as “debugger-first”, just to see what the experience was like.
Use C, tell gdb "-O0 -g -ggdb3"
Other ideas:
* Run the whole tree of processes under `gdb` with `set detach-on-fork off`.
* LD_PRELOAD a library that inserts the sleeps for you, maybe on startup or maybe on signal/exit.
Ideally, we'd have some kind of infrastructure to name and identify particular processes recursively.
It may not seem like it, but this is impressive progress. Getting a compiler to bootstrap at all is an accomplishment, especially for Rust since that depends on so many things working. Once it can reliably bootstrap, a lot of performance-improving steps can begin. Congrats!
Sounds like attempting to always inline a recursive function should be an error instead. But it's probably undesirable to make that change because it would likely break existing crates and thus backwards compatibility as well?
Really great read.
Someone mentioned recently that the slowness of rustc is in large part due to llvm. I know that is probably orthogonal to the work here, but I do like the idea of building the compiler with different toolchains, and that there may be follow on effects down the line.
Depends on the workload, but yes, codegen is a huge part of the total compilation time.
That said, that doesn't mean LLVM is always where the fixes need to be. For instance, one reason rustc spends a lot of time in LLVM is that rustc feeds more code to LLVM than it should, and relies on the LLVM optimizer to improve it. Over time, we're getting better about how much code we throw at LLVM, and that's providing performance improvements.
It's slow because the borrow checker is NP complete. LLVM may or may not generate slower code than GCC would for rustc, but I doubt it's anywhere close to the primary cause of the lack of snappy.
You're wrong it's been debunked that the borrow checker is any appreciable part of the compile time - Steve Klabnik actually verified it on here somewhere.
Edit: found it
https://news.ycombinator.com/item?id=44391240
I love vicarious engineering.
I just started playing with rust again today. Godspeed.