Summary Blocks - only show next post from within the block

I'm using summary blocks in 7.1 to pull in blog posts by category. When you click on the arrow at the bottom of the page, it takes you to the next blog post, by creation date, but does not take into account whether or not it's in that particular summary block. Is there a way to change this?

I can remove them entirely using 

Can you share link to page where you use summary block? We can check easier