Advances in assisted reproductive technology allow more and more infertile patients to benefit from in vitro fertilization (IVF) treatment. For an IVF treatment, selecting embryos that have developed into the blastocyst stage is a crucial step for the subsequent embryo transfer. In the clinic, it requires embryologists to keep observing the developmental status of embryos, which not only highly depends on the professional level of reproductive experts but greatly increases patients’ economic cost. To address this problem, we propose a novel task termed early blastocyst development prediction, which aims at predicting the potential that an embryo can develop into the blastocyst stage by partially observing its early development status so as to assist embryologists with early embryo selection. To achieve this goal, we collect a new IVF dataset with 2,898 time-lapse videos of embryo development. Based on this, we also propose a benchmark solution named Attentive Multi-focus Selection Network (AMSNet). Specifically, AMSNet is a deep learning-based time-lapse video analysis method that includes a separate attention mechanism to exploit the features of embryoscope images captured at multiple focal planes and a temporal feature channel shift operation to obtain memory capability over time-lapse videos. Experimental results demonstrate the effectiveness and clinical significance of our proposed AMSNet on our IVF dataset.