I want to sell my 325xi and get an E46 M3 but I am not sure if the trouble will be worth it.

how can I be sure the car I am being sold will be reliable and won't give me problems if its not a CPO? is carfax reliable way of checking history?

flip a coin...they are just expensive cars to own.

anymore expensive than a regular 3 series? and it what way.. maintenance or insurance or everything?

I heard that the engine has to be "hand tuned" but I am not sure what that means or if its even true

its a nice looking car, thats for sure

All of the M series cars by bmw are much more expensive and maintenance intensive to own. Main reason being that they are performance cars, where parts can wear quicker and with more power/harder driving things can break easier. All the basic stuff should be the same as the non-M counter part, but expect things like suspension, bushings, tires, etc to need replacement sooner. Plus the parts will prob cost more in general.

Theres no way to guarantee a used car is good or bad. Run a carfax, look at the car, drive the car, get a PPI. After that it's all luck. A car can be given a clean bill of health, and have the cooling system or something explode the next day. You can't always see problems coming, so preventative maintenance, although it may seem expensive, is often cheaper in the long run

Insurance will be much more expensive as well, and shop costs could be too mainly for the fact that I wouldn't take my M to just any normal shop. I would make sure they are very experienced or specialized in my specific car.

how can I be sure the car I am being sold will be reliable

Pre-purchase inspection by an impartial BMW shop.

and won't give me problems if its not a CPO?

Pre-purchase inspection by an impartial BMW shop.

is carfax reliable way of checking history?

No. Pre-purchase inspection at an impartial BMW shop.

Best couple hundred bucks you'll spend.