ExpressPrint: An Approach to Watermarking of Visual Foundation Models
https://doi.org/10.15514/ISPRAS-2025-37(6)-31
Abstract
The substantial cost of training from scratch of visual foundation models (VFMs) on large and vast datasets motivates the models’ owners to protect their intellectual property via ownership verification methods. In this work, we propose ExpressPrint, a novel approach to watermarking VFMs based on the fine-tuning of expressive layers of VFMs together with a small encoder-decoder network to embed the digital watermarks into a set of input images. Our method implies a small modification of expressive layers together with training an encoder-decoder neural network to extract user-specific binary messages from the hidden representations of certain input images. This method allows distinguishing between the foundation model provided to a user and independent models, thereby preventing unauthorized use of the model by third parties. We discover that the ability to correctly extract encoded binary messages from images transfers from a watermarked VFM to its functional copies obtained via pruning and fine tuning; at the same time, we experimentally show that non-watermarked VFMs do not share this property.
About the Authors
Anna Sergeevna CHISTYAKOVA
Ivannikov Institute for System Programming of the Russian Academy of Sciences,
Lomonosov Moscow State University
Russian Federation
Russian Federation
Senior Assistant at ISP RAS Research Center for Trusted Artificial Intelligence. Research interests: research and development of neural network architectures aimed at robustness to adversarial attacks.
Mikhail Aleksandrovich PAUTOV
Ivannikov Institute for System Programming of the Russian Academy of Sciences,
AIRI
Russian Federation
Russian Federation
Cand. Sci. (Phys.-Math.) in computer science, research scientist at AIRI and ISP RAS Research Center for Trusted Artificial Intelligence. His research interests include linear algebra, large deviations theory, certified robustness and digital watermarking.
References
1. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9.
2. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022). Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2), 3.
3. Ma, Jun, and Bo Wang. "Towards foundation models of biological image segmentation." Nature Methods 20.7 (2023): 953-955.
4. Schneider, Johannes, Christian Meske, and Pauline Kuss. "Foundation models: a new paradigm for artificial intelligence." Business & Information Systems Engineering 66.2 (2024): 221-231.
5. Uchida, Y., Nagai, Y., Sakazawa, S., & Satoh, S. I. (2017, June). Embedding watermarks into deep neural networks. In Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval (pp. 269-277).
6. Guo, J., & Potkonjak, M. (2018, November). Watermarking deep neural networks for embedded systems. In 2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD) (pp. 1-8). IEEE.
7. Li, Y., Zhu, L., Jia, X., Jiang, Y., Xia, S. T., & Cao, X. (2022, June). Defending against model stealing via verifying embedded external features. In Proceedings of the AAAI conference on artificial intelligence (Vol. 36, No. 2, pp. 1464-1472).
8. Pautov, M., Bogdanov, N., Pyatkin, S., Rogov, O., & Oseledets, I. (2024, August). Probabilistically robust watermarking of neural networks. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (pp. 4778-4787).
9. Quan, Y., Teng, H., Xu, R., Huang, J., & Ji, H. (2023). Fingerprinting deep image restoration models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 13285-13295).
10. He, Z., Zhang, T., & Lee, R. (2019). Sensitive-sample fingerprinting of deep neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4729-4737).
11. Peng, Z., Li, S., Chen, G., Zhang, C., Zhu, H., & Xue, M. (2022). Fingerprinting deep neural networks globally via universal adversarial perturbations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 13430-13439).
12. Sun M., Chen X., Kolter J. Z., and Liu Z. Massive Activations in Large Language Models. (2024) In First Conference on Language Modeling.
13. Dosovitskiy A., Beyer L., Kolesnikov A., Weissenborn D., Zhai X., Unterthiner T., Dehghani M., Minderer M., Heigold G., and Gelly S. (2020) An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2020.
14. Balestriero R., Ibrahim M., Sobal V., Morcos A., Shekhar S., Goldstein T., Bordes F., Bardes A., Mialon G., Tian Y., Schwarzschild A., Wilson A. G., Geiping J., Garrido Q., Fernandez P., Bar A., Pirsiavash H., LeCun Y., and Goldblum M. (2023) A Cookbook of Self-Supervised Learning. arXiv:2304.12210.
15. Chen T., Kornblith S., Norouzi M., and Hinton G. (2020) A simple framework for contrastive learning of visual representations. In International conference on machine learning, PmLR, 2020, pp. 1597–1607.
16. Caron M., Touvron H., Misra I., Jégou H., Mairal J., Bojanowski P., and Joulin A. (2021) Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9650–9660.
17. Radford A., Kim J. W., Hallacy C., Ramesh A., Goh G., Agarwal S., Sastry G., Askell A., Mishkin P., and Clark J. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, PmLR, 2021, pp. 8748–8763.
18. Oquab M., Darcet T., Moutakanni T., Vo H., Szafraniec M., Khalidov V., Fernandez P., Haziza D., Massa F., and El-Nouby A. (2024) DINOv2: Learning Robust Visual Features without Supervision, Trans. Mach. Learn. Res. J., 2024, pp. 1–31.
19. Xu, J., Wang, F., Ma, M., Koh, P. W., Xiao, C., & Chen, M. (2024, June). Instructional Fingerprinting of Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 3277-3306).
20. Song, H. J., Khayatkhoei, M., & AbdAlmageed, W. (2024). Manifpt: Defining and analyzing fingerprints of generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10791-10801).
21. Sander, T., Fernandez, P., Durmus, A., Douze, M., & Furon, T. (2024). Watermarking makes language models radioactive. Advances in Neural Information Processing Systems, 37, 21079-21113.
22. Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009, June). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (pp. 248-255).
23. Ecommerce Product Images for Product Categorization - https://www.kaggle.com/datasets/fatihkgg/ecommerce-product-images-18k, accessed 13.05.2025.
24. The Oxford-IIIT Pet Dataset - https://www.kaggle.com/datasets/tanlikesmath/the-oxfordiiit-pet-dataset, accessed 13.05.2025.
25. FoodSeg103 - https://huggingface.co/datasets/EduardoPacheco/FoodSeg103, accessed 13.05.2025.
26. Xu, T., Wang, C., Liu, G., Yang, Y., Peng, K., & Liu, W. (2024). United We Stand, Divided We Fall: Fingerprinting Deep Neural Networks via Adversarial Trajectories. Advances in Neural Information Processing Systems, 37, 69299-69328.
27. CLIP model for timm - https://huggingface.co/timm/vit_large_patch14_clip_224.openai, accessed 16.05.2025.
28. DINOv2 model for timm - https://huggingface.co/timm/vit_large_patch14_dinov2.lvd142m, accessed 16.05.2025.
29. Darcet, T., Oquab, M., Mairal, J., & Bojanowski, P. Vision Transformers Need Registers. In The Twelfth International Conference on Learning Representations.
Review
For citations:
CHISTYAKOVA A.S., PAUTOV M.A. ExpressPrint: An Approach to Watermarking of Visual Foundation Models. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2025;37(6):223-236. https://doi.org/10.15514/ISPRAS-2025-37(6)-31
Email this article 