Do you know how to resolve vectors into components? The most direct way to deal with this is to find separately the x-components and y-components for each of the three vectors (calling the magnitude of the unknown vector F for now), then find the sums of the x- and y-components.

You are asked to find the value of F that will make the magnitude of the vector sum smallest. The most direct way to do this requires calculus. Is your course calculus-based?