It's not the choice of wrong wheel size, just the max tyre width your bike will allow. Horses for courses narrow tyres can be a bit skittish off road esp on loose surfaces in you case a a 27.5" or 29 mtb would have been a better choice.
Done by experience of two bikes, one with thin tyres that cannot hack it by a big margin and the one with slightly thicker tyres that does. Loose surfaces, narrow gouges in paths, mud, pebbles, gravel, whatever surface the thicker tyres do the job and the thinner ones hopelessly fail.