Breaking an RSA-10 key (meaning an RSA key 10 bits long), requires that you try every prime number in the range of 2 to 100. There are 25 of those, meaning that the security of RSA-10 is about equivalent to a 5-bit symmetric cipher. Doubling that, going to RSA-20 means that to crack it, you need to try every prime number in the range of 2 to 1000. There are 168 of those so the security of RSA-20 is equivalent to about an 8-bit symmetric cipher. Doubling the key length did not give us the security we naively expected. Each subsequent bit adds to the security of the key but always less than the preceding bit. Thus we quickly reach a point of diminishing returns after which every bit adds so little that it is almost negligible.
That point is reached around RSA-2048 and adding more bits to that doesn't make much difference. For comparison, RSA-2048 gives about 112 bits of security and RSA-3072 gives roughly 128 bits of security.
When someone someone asks you whether it is worth using RSA-4096, the only added security compared to RSA-2048 is about 28 bits, meaning a total of 140 bits.
If you need more security than what RSA-2048 offers, consider using Elliptic Curve cryptography rather than using RSA with an ever increasing key length.
So how more secure could RSA be?
And the answer is: not much more secure.