While established predisposition genes account for syndromes like Cowden and Lynch syndrome, cancers frequently cluster in families without any identifiable pathogenic variant (PGV). We hypothesized that other germline factors beyond established PGVs might partially explain the large fraction of familial cancer cases with no known PGV. In this study, we systematically assessed potential genetic risk factors in 2,721 germline whole genomes from individuals with no recognized PGV, comparing 1,389 familial cancer cases across 18 cancer types to 1,332 controls with no family history of cancer in the NIH AllofUs Biobank. We performed genome-wide germline variant discovery using GATK-HC and GATK-SV to capture the full spectrum of rare and common SNVs, indels, and structural variants (SVs), notably including a total of 309,848 high-confidence SVs (9,639 SVs per genome). We first examined the rates of rare germline structural variants (SVs) predicted to cause loss-of-function of established predisposition genes, finding elevated rates of these likely pathogenic SVs in most cancer types compared to cancer-free controls, with neuroendocrine, colorectal, and ovarian cancer patients harboring notably high rates of these SVs (ORs: neuroendocrine 2.9; colorectal 3.1; ovarian 2.4). We also observed a small fraction of breast cancer patients carrying structural variants overlapping the BRCA1 promoter. We next tested for enrichment of rare missense variants of uncertain significance in established predisposition genes and found significant enrichment in endometrial, hematologic, prostate, and neuroendocrine cancer patients (p < 0.05). To search for potential new predisposition genes, we performed rare variant association tests for all 18,544 autosomal protein-coding genes, which nominated several candidate predisposition genes driven by rare damaging variants, such as BRAT1 in breast cancers (p=3.99e-5) and TSTD2 in thyroid cancers (p=3.26e-5). Finally, we assessed common-variant contributions and found that polygenic risk, long recognized in sporadic cancers, strongly contributed to familial cases in 11 of 15 cancer types with existing polygenic scores. Notably, polygenic risk scores were higher in patients from families with multiple occurrences of the same cancer type compared to patients whose own diagnosis was the only instance of that cancer in their family; conversely, in families with clusterings of a given cancer type, unaffected probands had polygenic risk scores for that cancer type on par with individuals who were both cancer-free and had no family history. Collectively, we estimate that 1-8.8% of unexplained familial cancers can be attributed to germline genetic factors detectable by genome sequencing but missed by routine gene panels, underscoring the potential clinical and diagnostic value of high-resolution germline testing for hereditary cancer patients and their families.