Most of the basic results on martingale problems extend to the setting in which the generator depends on a control. The ``control'' could represent a random environment, or the generator could specify a classical stochastic control problem. The equivalence between the martingale problem and forward equation (obtained by taking expectations of the martingales) provides the tool for extending linear programming methods introduced by Manne in the context of controlled finite Markov chains to general Markov stochastic control problems. The controlled martingale problem can also be applied to the study of constrained Markov processes (e.g., reflecting diffusions), the boundary process being treated as a control. Time permitting: the relationship between the control formulation and viscosity solutions of the corresponding resolvent equation will be discussed. Talk includes joint work with Richard Stockbridge and with Cristina Costantini.