There are a lot of factors involved in determining if a tree is too close to your house such as visible cracks the foundation, roof damage due to significant deposit of debris from the tree, poor air circulation, critical slopes, unstable soils and the general overall health and condition of the tree. The first thing that you would want to do is walk around the property to check for any visual signs that indicate a threat to the structures on the property. Visual signs would include but are not limited to cracks in trees, visual decay, fungus growing out of the tree trunk, cracks in the foundation. If any of these are found then you would want to have the tree evaluated more extensively.
If you are building a new house or structure you would want to talk with the county/city ordinance for more information on setbacks in that area. The Washington State Department of Natural Resources advises one tree length from the structure in rural areas, however there are other setbacks to consider as well.
In building a new structure if roots are cut, ripped or damaged due to excavation then that would be a good indicator that the tree is too close. If you have to trim all of the limbs off of the tree for a structure you may be promoting sun scorch which is the blistering of the tree trunk and promotes decay and rot. If there is a lot of fill required around the excavation site of the structure where trees remain it could deprive the tree from needed oxygen as well as promote declining health of the tree.