I suggest [The House on the Borderland][1] (1908) by William Hope Hodgson.

From the Wikipedia description (my bold indicates things that match the question): 

> Two men on **a two-week fishing vacation** in remote western Ireland are
> surprised to discover **a strange abyss**. On a rock spur above this pit
> they find ruins and buried in them a journal, which they read.

(and I summarize further information from the Wikipedia article:)

The journal recounts the experiences of a man who used to live in the now-ruined house, which had already been **abandoned for centuries** when he moved in.

One time **while he was reading**, he had a vision of **a horrible realm with monsters in it**. There were further visions on other occasions. Eventually he has a vision of himself and the house crumbling into dust.

The diary ends. The two fishermen ask other people and find corroboration of an old man who lived in the house, and the house eventually collapsed.


  [1]: https://www.goodreads.com/book/show/220937.The_House_on_the_Borderland?from_search=true&from_srp=true&qid=7jzjNz45Tn&rank=1