Designing wireless communication systems that efficiently utilize the resources frequency spectrum and electric power, leads to problems in mathematical optimization. Most of these optimization problems are difficult to solve because the objective functions are nonconvex. While some problems remain unsolved, the solutions proposed in the literature for the others are of somewhat limited use because the algorithms are either unstable or have too high a computational complexity. This dissertation presents several stable algorithms, most of which have polynomial complexity, that solve five different nonconvex optimization problems in wireless communication. Two centralized and two distributed algorithms deal with the power allocation that maximizes the throughput in the Gaussian interference channel (GIC)with various constraints. The most valuable of these algorithms, the one with the minimum rate constraints became possible after a significant theoretical development in the dissertation that proves that the throughput of the GIC has a new generalized convex structure called invexity. The fifth algorithm has linear complexity, and finds the power allocation that maximizes the energy efficiency (EE) of OFDMA transmissions, for a given subchannel assignment. Some fundamental results regarding the power allocation are then used in the genetic algorithm for determining the subchannel allocation that maximizes the EE. Pricing for channel subleasing for ad-hoc wireless networks is considered next. This involves the simultaneous optimization of many functions that are interconnected through the variables involved. A composite game, a strategic game within a Stackelberg game, is used to solve this optimization problem with polynomial complexity. For each optimization problem solved, numerical results obtained using simulations that support the analysis and demonstrate the performance of the algorithms are presented.