So, you will feel a little ‘play’ in the front forks, just because your handlebars are sitting on top of a long lever, which pivots on the headset, which is in that bit of the bike frame that you have photographed. Is there too much play in the headset? Difficult to say - there is a fine line between that bolt you picture being too tight or too loose. To tight and your steering will bind up and the bearings sitting at the top and bottom of the head tube will eventually fail. Too loose, and you get the motion in the forks/handlebars that you describe.
Is there a torque figure for that bolt in the picture? If you have that info, loosen that bolt, make sure that the forks are sitting correctly in the frame and the bearings are correctly in their cups, then tighten the bolt to the specified torque.