In this thesis, we propose a novel nonparametric modeling framework for financial time series data analysis, and we apply the framework to the problem of time varying volatility modeling. Existing parametric models have a rigid transition function form and they often have over-fitting problems when model parameters are estimated using maximum likelihood methods. These drawbacks effect the models' forecast performance. To solve this problem, we take Bayesian nonparametric modeling approach. By adding Gaussian process prior to the hidden state transition process, we extend the standard state-space model to a Gaussian process state-space model. We introduce our Gaussian process regression stochastic volatility (GPRSV) model. Instead of using maximum likelihood methods, we use Monte Carlo inference algorithms. Both online particle filter and offline particle Markov chain Monte Carlo methods are studied to learn the proposed model. We demonstrate our model and inference methods with both simulated and empirical financial data.