Unfortunately for you, there aren't all that many people on this site who are either geologists or in environmental remediation, so you're unlikely to get much in the way of field-specific information. That having been said, there are some more general techniques and insights that you might find useful.

The first thing is that no one is going to see your current experience as a black mark. It means that you have somewhat less experience in particular tasks. That means that if you're looking at a job focused around those tasks, you won't have as much experience in those tasks, and may need to settle for a position that doesn't fully reflect your number of years. This is potentially annoying, but as worst-case-scenarios go, it's not terrible... and when it comes time to move up to a higher-tier position, you can reach back and leverage your existing observation and management skills better.

The second thing is that you have specific niche experience as an auditor/overseer that is far beyond what a geologist of your years would normally have. As you've discovered, there is a market for entry-level and junior-level auditors/overseers. If you're happy to keep doing the sorts of things you've been doing, your resume and references are excellent for that, and the only question is whether or not that specific job market niche is large enough for you to consistently find jobs in it. Other similar roles might be as a direct assistant to a senior-level geologist who was fulfilling the standard senior-level role for a particularly large organization, as a budget pick for a government contractor who wanted some degree of oversight but didnt' want to pay for a more experienced geologist, and so forth.

The third thing is that you have been demonstrating a number of skills that employers tend to find generally useful and valuable. You've likely shown abilities in organization, generation of reports, establishing and maintaining customer relations, and so forth. Having a resume that leans heavily on those skills might get you hired for roles that are more on the administration side while petering off on the pure geology side, but it will get you hired. You're going to have to figure out how much that matters to you. Are you willing to write proposals for geology-themed contracts? What percentage of your time at work are you willing to have consist of writing proposals for geology-themed contracts?

Finally, there's the advice I give everybody, when you're looking for a job. Tell a story. Potential employers are going to look at your resume, look at your coverletter (if there is one), hear your spiel (if you have a chance to give one in person) and possibly stalk you online at least a little. From that, they're going to assemble in their head an idea of who you are as a person and as an employee. That idea is what they're going to base a hire/no-hire decision on. To get out ahead of that game, look at what is true about you, figure out a true story from that information that shows them that you are what they want to hire, and then make sure that the information you present sells that story. Based on what you've written here, I'd bet that you could easily tweak your resume description of your current job, and come up with at least three very different (all true) stories about what kind of a job it was, and what that implies about you as an employee (and, by extension, which employers might be interested in you for which positions). Of course, you shouldn't sell yourself to a job that you don't want. If you're telling a story about yourself, make sure that it's one you're willing to live if and when they take you.