Great question. That depends on the community rules in which you live. Most places require a zoning permit no matter the size of the structure, and a building permit if it is larger than a specific size. The first place you check is the local county or city planning department. Some have a handout addressing most of the key issues. The key question is how close you can build to the property line and other structures.
In some areas the Home Owners Association takes the place of the county zoning. So if you have a Home Owners Association, be sure and check your HOA rules first.
Same question if you’re a corner lot, have a septic system, LP tanks, lakes, rivers, high intensity power lines or neighborhood utility lines ( above or below ground). I found one site that even had approved plans on the books for a road going through a property, from the early 60’s. Basically you are asking about easements and setbacks. So do your homework.
For residential areas it is common to be limited to a shed size of 120 square feet, or less, to not require a building permit. In Virginia as of 2015, the state building code states that accessory structures 256 square feet or less do not need a building permit, the previous cutoff was 200 square feet. Individual counties have some time before they have to implement the new code, however most counties in Virginia implemented the code as soon as it came out. Comments and stories are welcome.