Generally speaking, video card performance can be judged by it's 'generation' and memory size
In this case I would say the Radeon card is superior (even though the Geforce has double the memory)
the ATI Radeon card is about 3(?) years old, the nVidia Geforce is 5 years old.
As far as I know, neither card can use dx10, but will work in Vista/7 with dx9, however
Software and support wise, IMO nvidia is better but some have had better luck with ati
Full disclosure: I've been a nVidia user for just under 10 years, I started out with ATI