Actually, the question how to test search engines is part of information retrieval (IR) problem. I will list just a few approaches how they can be evaluated/tested.

Traditional IR approach

One way in traditional IR is to prepare test setup, including:

Candidate queries

Dataset of indexed pages

Expected result set (included expected ranking) for each query

Next you obtain actual results by running your search engine over candidate queries and compare actual results with expected results using one of information retrieval metrics, e.g. precision/recall or nDCG. That would give you a quantitive answer how bad or good is your search engine with respect to expected result set (requirement).

Obviously, the answer will be context-specific. It will particularly depend on choice of candidate queries, they intention and wording, the dataset of indexed pages, and the way expected results were judged. So what was the context?

Goal-driven/Interactive IR approach

Another approach, maybe more practical, would be to involve potential users and let them assess quality of results for a given query. They could bring their own queries or you could define queries for them. Those could be a few people or a whole community, from which you could get more implicit feedback about results quality (e.g., results that are clicked are expected ones).

Problem isolation

Finally, the problem might be not to evaluate the overall quality of the search engine, but only narrow down the reason for low ranking of this particular query result. So, I would follow the problem you got:

You were told that a search engine result by target query does not satisfy a requirement by its position.

and asked myself:

What were the other results that were ranked higher than this?

Why were they ranked higher? Can I tell this from result snippets? Maybe they match query keywords but with a different meaning? So maybe the wording is incorrect? Would another wording of query intention improve the result?

The same query maybe worded differently, and the same wordings may have different query intention behind them. Do I really know what was intention of a searcher behind this query?

What precisely is the requirement? To boost higher sponsored pages? Or rank higher pages that matches query intention?

Those can help you reproduce the issue for other similar queries.