Fleshy-fruited plants depend fundamentally on interactions with frugivores for effective seed dispersal. Recent models of frugivory within spatially explicit networks make two general predictions regarding these interactions: rate of fruit removal increases (i.e., is facilitated) as densities of conspecific neighborhood fruits increase, and fruit removal rate varies positively with frugivore abundance. We conducted a field experiment that constitutes the first empirical and simultaneous test of these two primary predictions. We manipulated neighborhood abundances of arrowwood (Viburnum recognitum and Viburnum dentatum) fruits in southern New England’s maritime shrub community and monitored removal rates by autumn-migrating birds. Focal arrowwood plants in neighborhoods with high conspecific fruit density sustained moderately decreased fruit removal rates (i.e., competition) relative to those in low-density neighborhoods, a result that agrees with most field research to date but contrasts with theoretical expectation. We suggest the spatial contexts that favor competition (i.e., high-abundance neighborhoods and highly aggregated landscapes) are considerably more common than the relatively uniform, low-aggregation fruiting landscapes that promote facilitation. Patterns of arrowwood removal by avian frugivores generally varied positively with, and apparently in response to, seasonal changes in migratory frugivore abundance. However, we suggest that dense stands of arrowwood concentrated frugivore activity at the neighborhood scale, thus counteracting geographic patterns of frugivore abundance. Our results underscore the importance of considering spatial context (e.g., fruit distribution and aggregation, frugivory hubs) in plant-avian frugivore interactions.