I was wondering can't I just downloaded the packages from Debian and install them manually

Most Debian packages will not run on a Pi Zero. They do not support the ARMv6 CPU that it has. That's why we have Raspbian.

I'm running a Pi 3 B+ though my understanding is that the compiler on that system isn't able to compile due to it missing build options( to keep it compatible with earlier Pi's.) I don't know for sure it would install the host version on the target.

As this is your first time with buildroot, why not start building something that already exists and is known to work.
One example is noobs. https;//github.com/raspberry pi/noobs..
The initial build will take several hours as it builds all tools and libs, but thereafter it will be much quicker. You will need 6-8GB of free disk space.
Once you get that built and you understand it, you can start using buildroot for other projects.

As this is your first time with buildroot, why not start building something that already exists and is known to work.
One example is noobs. https;//github.com/raspberry pi/noobs..
The initial build will take several hours as it builds all tools and libs, but thereafter it will be much quicker. You will need 6-8GB of free disk space.
Once you get that built and you understand it, you can start using buildroot for other projects.

If you’re using buildroot to produce a root filesystem, Debian packages don’t really come into it. By default, it builds its own toolchain - this should be compatible as long as you make sure you tell it you’re using a Zero (start with the correct defconfig or configure it equivalently)

If you’re using buildroot to produce a root filesystem, Debian packages don’t really come into it. By default, it builds its own toolchain - this should be compatible as long as you make sure you tell it you’re using a Zero (start with the correct defconfig or configure it equivalently)

The problem is the compiler from what I've read you need one that's built for the pi 3 and the Raspbian stock is for pi 1 for maximum compatibility. I've got gcc 9 freshly compiled I'm going to give it shot when I home.

Reporting in I've been a little busy and my pi has been working hard compiling successfully Buildroot.

I tried it out and the base install (without setting anything up) works like I expected. Now I'm rebuilding with wireless network tools and more customization to my build.

I had one issue with my keyboard not being recognized and having to use a different one.

The compile is still going on I keep having small issues with the compile missing something and having to restart it after sorting it out however...

The bottom line is yes you can compile Buildroot on a RPI but it will take days! Just getting menuconfigs to build took 45-60 minutes for busybox and I went to bed waiting for the kernel one so 3 hours plus for that one. Buildroot's menuconfig is not such a wait.

I'll report back again after my customized version is built.edit
My last build has finished successfully. I had two problems with it one was running out of space on my drive easy to solve. The other was a missing build library libstdc++6 this was installed didn't have support for LIBCXX_3.4.26 I had to install the package libstdc++6_9.1.0-5_armhf.deb from the Debian Experimental branch. I just used the following command to install and it worked.