How to combine AR and VR in Unity?

I am currently programming an Android app with which you can create a village in AR mode on a plane. The user can touch the plane to get into the scene at this point. He can look around and can jump back into the AR mode by touching the screen.

How would you implement that? Currently I have 2 scenes: AR and VR. When clicking on the touch screen, the VR Scene is loaded and vice versa. The problem with this method is that the user needs to relocate the village in AR mode. With DontDestroyOnLoad the village is kept active but it is a bit buggy. Can I combine the two scenes? Would that make sense for the experience to be more immersive? How would you do that? 2 cameras (AR and non-AR) and then only one activate?!

Thanks! Kalisser

     void Update()
{
    if (Input.touchCount > 0 && Input.GetTouch(0).phase == TouchPhase.Began)
        {
            m_prefab = null;
            Ray ray = Camera.main.ScreenPointToRay(Input.GetTouch(0).position);
            RaycastHit hit;

            if (Physics.Raycast(ray, out hit, Mathf.Infinity))

            {
            Instantiate(m_prefab);
            }
        }
    }
public void SetCameraAtPosition()
{
    if (SceneManager.GetActiveScene().name == "VR")
    {
        Camera.main.transform.position = cameraPosition;
    }
}