Quantifying evolutionary dynamics of cancer initiation and progression can provide insights into more effective strategies of early detection and treatment. Here we develop a mathematical model of colorectal cancer initiation through inactivation of two tumor suppressor genes and activation of one oncogene, accounting for the well-known path to colorectal cancer through loss of tumor suppressors APC and TP53 and gain of the KRAS oncogene. In the model, we allow mutations to occur in any order, leading to a complex network of premalignant mutational genotypes on the way to colorectal cancer. We parameterize the model using experimentally measured parameter values, many of them only recently available, and compare its predictions to epidemiological data on colorectal cancer incidence. We find that the reported lifetime risk of colorectal cancer can be recovered using a mathematical model of colorectal cancer initiation together with experimentally measured mutation rates in colorectal tissues and proliferation rates of premalignant lesions. We demonstrate that the order of driver events in colorectal cancer is determined primarily by the fitness effects that they provide, rather than their mutation rates. Our results imply that there may not be significant immune suppression of untreated benign and malignant colorectal lesions.