If you're short of money, too, you will want to avoid impulse buys and only get what's needed, at the right time. Every horse is different, and you may find after time and experience that you want to try something different, or more complicated. But, for the basic training of horse and rider, this is what you will need.
The first year:
Shopping list: Grooming kit, including hoofpick and soft bodybrush; good-quality halter and lead rope.
This is a cheap year for equipment! You will be mainly spending your time getting to know your new baby, and helping him to get to know you. All you need to do at this stage is handle him frequently, teach him to lead and lift his feet (see my article First Steps for Foals for more details).
The second year:
Shopping list: Bridle; bit; reins; saddle; numnah or saddle cloth; girth
This is the year for teaching your horse respect for you and other people. He will be continuing to grow, so you will not be riding him yet - but this is the year you should be buying his first tack, and letting him become absolutely familiar with it before you have to use it.
Make sure your bridle is good, strong leather, with the best stitching and craftwork you can afford. It must fit perfectly: if you are not sure about the size to buy, or about how to fit it, seek expert advice from a saddler or from a qualified equestrian professional.