There is some method to the madness for how credit card numbers get assigned or generated. Most payment networks follow this type of numbering convention, though American Express is a notable exception.
Card numbers are commonly called Primary Account Numbers (PAN) or Financial Primary Account Numbers (FPAN). Their format is standardized by ISO/IEC 7812. Discover, Mastercard, and Visa card numbers are typically 16 digits, while American Express card numbers are typically 15 digits. Originally, the ISO standard specified that the first 6 digits of a PAN are the Issuer Identification Number (IIN) or Bank Identification Number (BIN), nicknamed BIN6. More recently, to free up additional card numbers, the ISO standard moved to BIN8, where the first 8 digits are fixed for the account range. The last card number digit is the Luhn check digit, and the number between the BIN and the last digit is the (internal) account number.
Traditionally, the first digit of a card number indicates the payment network:
Here is the breakdown for a (made-up) Mastercard card number 5120 9722 9283 3716:
Card numbers aren't so random after all. With a BIN8, that leaves 16 - 8 - 1 (check digit) = 7 digits to change, for 107 = 10,000,000 = 10 million card numbers. With a BIN6, the number of available card numbers rises to 109 = 1,000,000,000 = 1 billion card numbers. This illustrates why the ISO standard shifted to BIN8 - a small credit union could use two BIN6s, one for their credit portfolio and one for their debit portfolio. Even though they may only issue 20,000 PANs in 10 years, that small credit union is blocking 2 billion PANs by sitting on a BIN6. Giving them two BIN8s still gives them more than enough PANs while keeping many other valid card numbers available for issuing. Even though 10 million PANs sounds like a lot of potentially available card numbers, they can run out faster than you may expect. Card numbers can be unavailable or used up if a card is replaced, replaced for fraud, if a customer closes their account, etc. Tokenization, which is being heavily pushed by nearly all players in the card ecosystem, can take up several unique card numbers, too. (If you have a card in Apple Pay, tap on it from the Wallet app, tap the three dots in a circle, tap card number, and you'll see that the Physical Card last four digits are different from the Apple Pay last four digits.) You can see how 10 million card numbers suddenly doesn't seem quite so impossible to hit.
That leads to fraud itself - it is easy to procedurally generate card numbers in any programming language if you know or guess the BIN6 or BIN8. Alternatively, fraudsters can pay a third party to test potential card numbers or to hack any number of providers or vendors who store card information, potentially insecurely. A dirty secret in the transaction authorization world is that even though you may type in your card number, three- or four-digit code, and entire billing address into an online form, the merchant may only send the PAN to seek an approval. There are benefits to sending and checking additional information, but not all merchants take advantage of those benefits. Why is only a PAN required for a transaction? Because we have, as an ecosystem, made choices on the tradeoffs between accessibility (ease of use/ease of payment) and security (preventing card fraud). Surprisingly, the optimal amount of fraud is non-zero. Having zero fraud means your system is so locked down as to make it nearly impossible to use, or it's so small that it doesn't matter. Especially in the United States, we accept the tradeoff of higher fraud in exchange for higher approval rates and more approved transactions - more interchange revenue for issuing banks and more sales for merchants.
So, a merchant doesn't want to check any other information besides the PAN? That's fine, but that merchant may incur more liability for fraudulent transactions. Fraudsters could also hack or create a fake merchant to use in card testing. At large scale, enumerating on BINs to figure out "live" PANs is called a BIN attack. Banks know about BIN attacks and have tools and processes to prevent and deflect them, but small-scale testing or old-fashioned card skimming could still occur and compromise a PAN. A fraudster could generate your exact card number, test it at a shoddy or compromised merchant, get a hit, and be off to the races. Some banks that perform mass issuance or re-issuance of card numbers can run into problems, too. If the bank establishes sequential PANs (e.g. xx0010, xx0020, etc) and/or sets the card expiration date to be five years from issuance, it's easy for a fraudster to determine additional information. Wells Fargo's Bilt card portfolio was a notable example of persistent BIN attacks based on issuing sequential PANs and hard-coding in expiration dates, as opposed to randomly generating an expiration date. Just search for Bilt card fraud.
That is how a card could never leave your wallet but still experience fraud. Fortunately, strong consumer protections in the United States should limit your liability. Why is the card numbering scheme so easy to figure out? Because that's how it was created decades ago, and we collectively have not moved away from it. Tokenization becomes more and more important because it adds additional security to card numbers that may seem more secure than they actually are. Even tokenization is transforming: going from a different PAN (Apple Pay virtual card example) to non-PAN text that still effectively masks and tokenizes the underlying payment account. It is very possible in the near future that cards will not have the magnetic stripe and will not have a printed or embossed card number. The only "card present" ways to pay with a card will be tapping the card or inserting the chip into the payment terminal, and the only online or "card not present" ways to pay will involve digital wallets like Apple Pay, or another link to a bank account to share tokenized PAN information without exposing the actual PAN. (That's why the big banks who co-founded Zelle are pushing a new digital wallet called Paze.) Pretty cool, right?
If you are interested in reading more about cards and the payment card system, take a look at Patrick McKenzie's Credit cards as a legacy system and Improving how credit cards work under the covers.