Modern technologies generate vast amounts of fine-grained data at an unprecedented speed.Nowadays, high-dimensional data, where the number of variables is much larger than thesample size, occur in many applications, such as healthcare, social networks, and recommendation systems, among others. The ubiquitous interest in these applications has spurredremarkable progress in the area of high-dimensional data analysis in terms of point estimation and computation. However, one of the fundamental inference task, namely quantifyinguncertainty or assessing statistical significance, is still in its infancy for such models. In thefirst part of this dissertation, we present efficient procedures and corresponding theory forconstructing classical uncertainty measures like confidence intervals and p-values for singleregression coefficients in high-dimensional settings.In the second part, we study the compressed sensing reconstruction problem, a well known example of estimation in high-dimensional settings. We propose a new approachto this problem that is drastically different from the classical wisdom in this area. Ourconstruction of the sensing matrix is inspired by the idea of spatial coupling in codingtheory and similar ideas in statistical physics. For reconstruction, we use an approximatemessage passing algorithm. This is an iterative algorithm that takes advantage of thestatistical properties of the problem to improve convergence rate. Finally, we prove thatour method can effectively solve the reconstruction problem at (information-theoretically) optimal undersampling rate and show its robustness to measurement noise.