Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks
Collaboration of LMs and SMs for domain tasksAbstract
Large language models support broad generalization but often require substantial data and computation for domain tasks. Small models are efficient and domain-specific, but they have limited general coverage. This survey studies collaboration between large and small models for private-domain adaptation. It focuses on cross-boundary settings where models and data are held by different parties, creating constraints on privacy, security, integrity, and resources. The paper organizes prior work by information flow: downward knowledge transfer from large models to small models, upward transfer from small models to large models, and inference-time collaboration across parties. It further identifies deployment challenges and frames practical collaboration as a multi-objective optimization problem.
