{"id":{"repo_id":"vt","oai_identifier":"oai:vtechworks.lib.vt.edu:10919/135566"},"canonical_url":"https://search.dev.ndltd.org/etd/vt/oai:vtechworks.lib.vt.edu:10919/135566","repository":{"repo_id":"vt","name":"Virginia Tech","base_url":"https://vtechworks.lib.vt.edu/oai/request"},"display":{"title":"Model Control through Lightweight Activation Steering for Vision Language Models","abstract":"This work introduces SteerVLM, a lightweight steering module designed to guide Vision- Language Models (VLMs) towards outputs that better adhere to desired instructions. Our approach learns from the latent embeddings of paired prompts encoding target and converse behaviors to dynamically adjust activations connecting the language modality with image context. This provides fine-grained, inference-time control over complex output semantics without modifying model weights while preserving performance on off-target tasks. Our steering module requires learning parameters equal to 0.14% of the original VLM's size. Additionally, our steering module gains model control via dimension-wise activation mod- ulation and adaptive layer-wise steering without requiring pre-extracted static vectors or manual tuning of intervention points. Furthermore, we introduce VNIA (Visual Narrative Intent Alignment), a multimodal dataset specifically created to facilitate the development and evaluation of VLM steering techniques. Our method outperforms existing intervention techniques on steering and hallucination mitigation benchmarks for VLMs and proposes a robust solution for multimodal model control through activation engineering.","abstract_html":"This work introduces SteerVLM, a lightweight steering module designed to guide Vision- Language Models (VLMs) towards outputs that better adhere to desired instructions. Our approach learns from the latent embeddings of paired prompts encoding target and converse behaviors to dynamically adjust activations connecting the language modality with image context. This provides fine-grained, inference-time control over complex output semantics without modifying model weights while preserving performance on off-target tasks. Our steering module requires learning parameters equal to 0.14% of the original VLM's size. Additionally, our steering module gains model control via dimension-wise activation mod- ulation and adaptive layer-wise steering without requiring pre-extracted static vectors or manual tuning of intervention points. Furthermore, we introduce VNIA (Visual Narrative Intent Alignment), a multimodal dataset specifically created to facilitate the development and evaluation of VLM steering techniques. Our method outperforms existing intervention techniques on steering and hallucination mitigation benchmarks for VLMs and proposes a robust solution for multimodal model control through activation engineering.","abstract_has_math":false,"creators":["Sivakumar, Anushka"],"institution":"Virginia Tech","degree_name":"Master of Science","degree_level":"masters","degree_discipline":"Computer Science & Applications","degree_department":"Computer Science and#38; Applications","school":null,"contributors":[],"advisors":[],"committee_chairs":["Thomas, Christopher Lee"],"committee_members":["Wang, Xuan","Yanardag Delul, Pinar"],"year":2025,"date_issued":"2025-06-23","date_published":"2025-06-23","updated_at":"2026-07-24T05:56:18Z","subjects":["Vision Language Models","Activation Engineering","Steering","Large Language Models","Latent Space Arithmetic","Multimodal."],"languages":["en"],"rights":["In Copyright"],"rights_urls":["http://rightsstatements.org/vocab/InC/1.0/"],"identifier_entries":[{"key":"dc:identifier.other","label":"Dc Identifier Other","values":["vt_gsexam:44270"],"render_values":[{"text":"vt_gsexam:44270","href":null,"code":true}]}]},"links":{"outbound_url":"https://hdl.handle.net/10919/135566","outbound_label":"Handle","outbound_source":"dc:identifier.uri"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.committeechair","label":"Committee Chair","values":["Thomas, Christopher Lee"]},{"key":"dc:contributor.committeemember","label":"Committee Member","values":["Wang, Xuan","Yanardag Delul, Pinar"]},{"key":"dc:contributor.department","label":"Department","values":["Computer Science and#38; Applications"]},{"key":"dc:creator","label":"Author","values":["Sivakumar, Anushka"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2025-06-24T08:01:03Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2025-06-24T08:01:03Z"]},{"key":"dc:date.issued","label":"Date","values":["2025-06-23"]},{"key":"dc:publisher","label":"Institution","values":["Virginia Tech"]},{"key":"dc:type","label":"Dc Type","values":["Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science & Applications"]},{"key":"thesis:degree_level","label":"Degree Level","values":["masters"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Master of Science"]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["Virginia Polytechnic Institute and State University"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Vision Language Models","Activation Engineering","Steering","Large Language Models","Latent Space Arithmetic","Multimodal."]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language.iso","label":"Language (ISO)","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["In Copyright"]},{"key":"dc:rights.uri","label":"Rights URI","values":["http://rightsstatements.org/vocab/InC/1.0/"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.other","label":"Dc Identifier Other","values":["vt_gsexam:44270"]},{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://hdl.handle.net/10919/135566"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["This work introduces SteerVLM, a lightweight steering module designed to guide Vision- Language Models (VLMs) towards outputs that better adhere to desired instructions. Our approach learns from the latent embeddings of paired prompts encoding target and converse behaviors to dynamically adjust activations connecting the language modality with image context. This provides fine-grained, inference-time control over complex output semantics without modifying model weights while preserving performance on off-target tasks. Our steering module requires learning parameters equal to 0.14% of the original VLM's size. Additionally, our steering module gains model control via dimension-wise activation mod- ulation and adaptive layer-wise steering without requiring pre-extracted static vectors or manual tuning of intervention points. Furthermore, we introduce VNIA (Visual Narrative Intent Alignment), a multimodal dataset specifically created to facilitate the development and evaluation of VLM steering techniques. Our method outperforms existing intervention techniques on steering and hallucination mitigation benchmarks for VLMs and proposes a robust solution for multimodal model control through activation engineering."]},{"key":"dc:description.abstractgeneral","label":"General Abstract","values":["This research introduces a way to improve how AI models, especially those that understand both images and language, follow general concepts. We've developed a simple system that guides these models to behave in accordance with desired concepts. Our system uses pairs of contrasting examples to teach the AI what we're looking for. It's designed to work within the AI's internal processes, allowing us to control complex behaviors like steering towards a particular topic or sentiment, and ensuring the output aligns with a specific viewpoint. Importantly, this system is easy to use and doesn't negatively affect the AI's other abilities. A key advantage is that it automatically figures out the best way to guide the AI, without needing special adjustments. In addition to our proposed steering module, we introduce a novel Vision-Language dataset, VNIA (Visual Narrative Intent Alignment), designed to facilitate the development and evaluation of vision-language model steering."]},{"key":"dc:description.degree","label":"Dc Description Degree","values":["Master of Science"]},{"key":"dc:format.medium","label":"Dc Format Medium","values":["ETD"]},{"key":"dc:title","label":"Title","values":["Model Control through Lightweight Activation Steering for Vision Language Models"]}]}],"canonical_facts":{"dc:contributor.committeechair":["Thomas, Christopher Lee"],"dc:contributor.committeemember":["Wang, Xuan","Yanardag Delul, Pinar"],"dc:contributor.department":["Computer Science and#38; Applications"],"dc:creator":["Sivakumar, Anushka"],"dc:date.accessioned":["2025-06-24T08:01:03Z"],"dc:date.available":["2025-06-24T08:01:03Z"],"dc:date.issued":["2025-06-23"],"dc:description.abstract":["This work introduces SteerVLM, a lightweight steering module designed to guide Vision- Language Models (VLMs) towards outputs that better adhere to desired instructions. Our approach learns from the latent embeddings of paired prompts encoding target and converse behaviors to dynamically adjust activations connecting the language modality with image context. This provides fine-grained, inference-time control over complex output semantics without modifying model weights while preserving performance on off-target tasks. Our steering module requires learning parameters equal to 0.14% of the original VLM's size. Additionally, our steering module gains model control via dimension-wise activation mod- ulation and adaptive layer-wise steering without requiring pre-extracted static vectors or manual tuning of intervention points. Furthermore, we introduce VNIA (Visual Narrative Intent Alignment), a multimodal dataset specifically created to facilitate the development and evaluation of VLM steering techniques. Our method outperforms existing intervention techniques on steering and hallucination mitigation benchmarks for VLMs and proposes a robust solution for multimodal model control through activation engineering."],"dc:description.abstractgeneral":["This research introduces a way to improve how AI models, especially those that understand both images and language, follow general concepts. We've developed a simple system that guides these models to behave in accordance with desired concepts. Our system uses pairs of contrasting examples to teach the AI what we're looking for. It's designed to work within the AI's internal processes, allowing us to control complex behaviors like steering towards a particular topic or sentiment, and ensuring the output aligns with a specific viewpoint. Importantly, this system is easy to use and doesn't negatively affect the AI's other abilities. A key advantage is that it automatically figures out the best way to guide the AI, without needing special adjustments. In addition to our proposed steering module, we introduce a novel Vision-Language dataset, VNIA (Visual Narrative Intent Alignment), designed to facilitate the development and evaluation of vision-language model steering."],"dc:description.degree":["Master of Science"],"dc:format.medium":["ETD"],"dc:identifier.other":["vt_gsexam:44270"],"dc:identifier.uri":["https://hdl.handle.net/10919/135566"],"dc:language.iso":["en"],"dc:publisher":["Virginia Tech"],"dc:rights":["In Copyright"],"dc:rights.uri":["http://rightsstatements.org/vocab/InC/1.0/"],"dc:subject":["Vision Language Models","Activation Engineering","Steering","Large Language Models","Latent Space Arithmetic","Multimodal."],"dc:title":["Model Control through Lightweight Activation Steering for Vision Language Models"],"dc:type":["Thesis"],"thesis:degree_discipline":["Computer Science & Applications"],"thesis:degree_level":["masters"],"thesis:degree_name":["Master of Science"],"thesis:institution_name":["Virginia Polytechnic Institute and State University"]},"updated_at":"2026-07-24T05:56:18Z"}