buffer overflow

{   
  char buf[8];
  sprintf(buf,"AAAA%3s","XXXXXXXX");
  printf("%s/n",buf);
}

what will happen?

The buffer have 8 characters space and only 3 free characters left, however, "XXXXXXXX" is 8 characters long.

It makes a lot of sense to consider what happens in your and, more importantly, similar, cases. As other posters have noted, it invokes UB. That's probably true. However, the world does not stop simply because someone did not define what exactly should happen next. And what physically happens next, may well be a major security hole.

If your string XXX... comes from uncontrolled sources, you are very close to generating a buffer overflow vulnerability.

(1) Your stack typically "grows" backwards, i.e. the smaller the addresses, the more the stack is filled.

(2) Strings expect the characters belonging to that string to be stored so that character n+1 is stored after character n.

(3) When you call a function, the return address, i.e. the address of the instruction that is to be executed after the function returns, is pushed to the stack (among other things, typically).

Now consider a stack frame of your function.

|----------------|
| buf [size 8]  |
|----------------|
| (func args)  |
|----------------|
| (other stuff) |
|----------------|
| return address |
|----------------|

By finding out what exactly the offset between buf and the return address on the stack is, a malicious user may manipulate input to your application in a way that the XXX... string contains an address of the attacker's choosing at just the point where the uncontrolled sprintf function will overwrite the return address on the stack. (NB: Better use snprintf if it's available to you). Thereby the attacker mounted a buffer overflow attack. He might use something like the NOP sled technique to have your application start a shell for him. If you were writing an application that ran under a privileged user account, you'd just have provided an attacker with a first-grade entry to your costumer's system, an ACE hole, if you will.

Update

The run-time error you experience may well be due to an overwritten return address. Since you filled it with, basically, gargabe, the address the CPU jumped to did probably contain byte sequences that, interpreted as program text, cause an invalid memory access (or the address itself was already bad).

It should be noted that some compilers can help against these kinds of errors. GCC, for example, has the -fstack-protector. I'm not familiar with how good those features are.