One way to balance is contract testing where you run a test suite against your mock/stub and the same test suite against the real thing. Your mock/stub would usually just have hardcoded data that aligns with the test suite to pass.
Another way is creating separate code libraries to abstract away external services. For instance, you might write ObjectStorage library that has real integration tests that use S3 but in your application you just stub out the ObjectStorage library. A lot of frameworks have verifying doubles or "magic mocks" or similar that will stub the method signatures so that can help make sure you're calling methods correctly (but wouldn't help catch data/logic issues).
Live upstreams are also prone to inconsistent state and race conditions--especially when you try to run more than 1 instance of the test suite at once.
For web services, I also like using "fixtures". You just manually run against a real service and copy the requests to a file (removing any sensitive data) than have your stub load the data from the file and return it. If you hit any bugs/regressions, you can just copy the results that caused them into a second file and add another test case.
Another way is creating separate code libraries to abstract away external services. For instance, you might write ObjectStorage library that has real integration tests that use S3 but in your application you just stub out the ObjectStorage library. A lot of frameworks have verifying doubles or "magic mocks" or similar that will stub the method signatures so that can help make sure you're calling methods correctly (but wouldn't help catch data/logic issues).
Live upstreams are also prone to inconsistent state and race conditions--especially when you try to run more than 1 instance of the test suite at once.
For web services, I also like using "fixtures". You just manually run against a real service and copy the requests to a file (removing any sensitive data) than have your stub load the data from the file and return it. If you hit any bugs/regressions, you can just copy the results that caused them into a second file and add another test case.