As I understand, CPU is central processing unit which I can program it to do some works, and GPU is something you can program it do do some works, in parallel but its function is limited, e.g it can add but can not multiply. But as all computer world is based on binary bits and logic functions AND, OR, NOT, so if one tries, he can program GPU to do whatever CPU can do, right? Or there are works that CPU can do but GPU can not do?

The CPU is often a chip on the motherboard under a huge heatsink. The GPU is a chip (also under a heatsink) on an expansion card sitting on a PCI-e, AGP or PCI slot on the motherboard.

The CPU is where everything runs, windows etc. All programs also runs on the CPU. The GPU is an "extension" to the CPU. The GPU and CPU communicate using a bus, most often the PCIe bus. The CPU has its system memory and the GPU has its device memory.

The GPU is optimized for doing many calculations using floats. For this reason data is sent to the GPU memory to do a huge number of float operations. The bonus is that the video input to your monitor is hooked up to the GPU memory such that you can generate animation in real time on the GPU without tranferring the result back to system memory.

The GPU and CPU can do most of the same math operations, but the GPU is optimized for floats.

The CPU is indeed the Central Processing Unit, commonly referred to as 'the processor' of your computer. The GPU is the Graphics Processing Unit, which is a processor chip found on your graphics card.

The CPU is designed for general purpose usage and depending on its architecture it can provide security features (like protection rings in the x86 architecture), different operating modes and a whole bunch of possible extensions.

The GPU was originally designed solely for doing graphics-related jobs and can process a huge amount of floating point operations in comparison to your standard CPU. I don't know where you got your information about available operations on GPUs, but they can definitely do multiplication (would be quite terrible if they weren't able to do something this trivial).

These days the GPU is used more and more for general purpose tasks which involve heavy number crunching, but generally speaking a GPU's ISA (Instruction Set Architecture) is much more limited than your common CPU's ISA.

You don't write whole programs for GPU. What you do is that you have it run a small set of instructions (a kernel) in paralell many times over on a set of data.

For instance, you copy a buch of vertex data to the device memory, then ask the GPU to do math on the vertices. Then for each set of vertices you output a bunch of pixels which you also run a kernel on each pixel. These are called vertex and pixel shaders and are used to generate an image to be displayed on your monitor.

Recently, people started to use this pipeline for other purposes then generating an image. Namely number crunshing.

Yes, modern GPUs are able to do branching, but I believe CPUs are generally more efficient at doing branches, or at least it used to be like that.

You have to keep in mind that a GPU is designed to do a large amount of similar jobs at once in parallel, that's why it's so efficient at doing heavy calculations. It'd be extremely hard or maybe even impossible to build an operating system kernel which could run on this kind of architecture. Working with memory would also be a major issue.

About ray tracers, there are a lot of ray tracing implementations which run on the GPU or which at least use the GPU to accelerate the process. The problem with CPU - GPU interop is that you'll encountering latency issues when uploading data to the GPU or when reading back data. It's only worth it to upload a job to the GPU if the speed improvement you'll get from it takes into account the memory latency from the data upload and potential readback as well.

GPUs have been Turing complete for a while, so they can theoretically perform any calculation a CPU can perform.
In practice, the GPUs don't have any way to communicate with peripherals, just with the CPU, so that should make it obvious why one can't run a computer with a GPU alone.
It would also be extremely impractical to run some types of programs on a GPU - for example, a general purpose operating system - because the GPUs lack key features such as interrupts that are critical to implementing those programs in practice.

The GPU has had to execute both the 'if' and the 'else', but ignoring some parts of it's arrays for each branch, and merging the results at the end. This is wasteful -- e.g. say the GPU has the capability to work on 3 bits of data at once, in this example it's only working on 1 or 2 bits of data at once.The more nested branches you add, the more wasteful this becomes... so those kinds of programs are better of running on regular CPU's (or being redesigned to better suit this style of hardware).

In practice, the GPUs don't have any way to communicate with peripherals, just with the CPU... the GPUs lack key features such as interrupts that are critical to implementing those programs in practice.

Out of interest's sake, GPUs can generate CPU interrupts and write to arbitrary addresses (which might be mapped to peripherals), but these abilities aren't exposed on PC's (outside of the driver).

GPU does the math too. I would say it's more like, CPU does the logic that brings everything together, and GPU is the powerhouse that drives rendering, and possibly physics (and perhaps other stuff too).

“If I understand the standard right it is legal and safe to do this but the resulting value could be anything.”

GPU does the math too. I would say it's more like, CPU does the logic that brings everything together, and GPU is the powerhouse that drives rendering, and possibly physics (and perhaps other stuff too).

Also audio processing can be done on the GPU

C dominates the world of linear procedural computing, which won't advance. The future lies in MASSIVE parallelism.

If you want to learn how to write graphical programs for a GPU, you'll use HLSL (via Direct3D), GLSL (via OpenGL), or Cg (via either).However, you don't write full programs in these languages -- you only write small functions (such as to colour in a pixel, or move a vertex) which are called by Direct3D or OpenGL, so you also have to learn how to write D3D/GL programs on the CPU as usual.

The small GPU programs/functions are called "shaders", and GLSL/HLSL are "shader languages".Your GPU supports Direct3D 10.1 and OpenGL 3.3, which both in turn support fairly modern versions of the above shader languages.

If you want to learn how to use the GPU for non-graphical uses, then you're in the same boat, except the CPU-side APIs that you first need to learn are (one of) DirectCompute, OpenCL or CUDA.

May I ask why you wish to program onto the GPU directly? Chances are if you have an actual reason (and know this reason etc) then you already have knowledge of the GPU surpassing most of us here which (no offense intended here) it is obvious you do do not.

The actual lettering GPU stands for Graphics Processing Unit. This shows what its intended for pretty well.
Usually it has many many many cores (hundreds on several gaming cards, my own has over 300), these cores are capable of floating point maths natively (many CPU's are not and instead have additional circuitry for that). 1 processor core can only run a single task at a time, any code you ever write will also only ever use 1 core at a time (even GPU programming I believe) unless you specifically program it to make use of all the cores.
Something like a program that just takes 2 numbers, adds them together and spits the result back out will have no benefit to being parallelised. Something like graphics where you need to be doing a million things at once will see a huge benefit.
A single GPU core isn't actually very powerful with the exception of its floating point abilities, infact it wouldn't surprise me if the primary core of some of the higher end smart phones is more powerful (although I admit, I did not go online and check that). In theory as a GPU is what is referred to as turing complete it can do anything a CPU can, in practice a GPU has no way of interfacing with things like your keyboard etc, it can only spit out video data and only when its told to. In theory linux or something could be compiled to run on a GTX560, in practise though it would be running on just one of its several hundred cores with a ridiculous amount of limitations.

Some other tasks now done on the GPU are physics/biology/chemistry simulations and certain mathematical problems on large amounts of data although per core may be slower to do on the GPU than the CPU can of course be done on 100 or more pieces of data at once unlike your CPU where you can hope to do it on 4 at most maybe.

OpenCL can be hardware accelerated on both AMD and NVidia GPU's (aswell as some other more obsure hardware) or it can be run in pure CPU mode. NVidia have their CUDA library which works on NVidia GPU's and PPU's only (PPU's are rare and I don't think NVidia actually make them any more), for some things CUDA is faster, some OpenCL is faster, NVidia's PhysX physics library can use CUDA when present. The Bullet physics library can optionally use OpenCL or CUDA when possible too. OpenCV can use OpenCL aswell.

Generally any purpose you will need hardware acceleration from yourself will have a library available. GPU programming is a highly specialised field. For the majority of this post I am ignoring programmable shader pipelines because although they are still technically programming on a GPU it isn't what you appear to be after.

I am learning about ray tracing because I love it, so I am trying to write on GPU (cast many rays at once) to test its limit. It is not my main job just hobby but I want to understand the concept. I am leaning to directcompute now but it seems tutorial on this field is very limited.

GPU does the math too. I would say it's more like, CPU does the logic that brings everything together, and GPU is the powerhouse that drives rendering, and possibly physics (and perhaps other stuff too).

Perhaps the best way to say it is that a CPU is a generalized processor, and a GPU is a processor designed to efficiently operate on graphical entities. It's more or less a very specialized CPU - what it is designed to do, it does much faster than a general CPU, and what it wasn't designed to do, it does much, much, much slower.

... and a GPU is a processor designed to efficiently operate on graphical entities.

The rasterizer is pretty much the only thing that's really graphics-centered in GPUs anymore, they're good at all problems that can be solved in a massively parallel manner and with similar loop counts etc (little branching)