Phase sensitive optical parametric amplification (PS-OPA) can realize noiseless amplification in theory. However, Raman effect will affect the four-wave mixing (FWM) process and induce excess noise for fiber-based PS-OPA systems with wide bandwidth. To achieve a more precise description of the gain evolution within the PS-OPA and its associated noise characteristics, we established a more comprehensive theoretical model to analyze the joint impact of the Raman effect and the FWM effect. It is found that the competition between Raman effect and FWM effect can change the evolution trend of power and relative phase, thus affect gain and noise figure (NF) features of the PS-OPA. For signal waves with different frequency shift from the pump, obvious asymmetry for both gain and NF spectra is induced by Raman scattering. This model can be applied to investigate frequency non-degenerate PS-OPA with third-order nonlinear media such as highly nonlinear optical fibers (HNLF) and silicon waveguides.