Securing SFM systems using Generative Diffusion Modeling

Author(s): Ashwin Pokharel

Mentor(s): Bo Han, Computer Science

Abstract
Point clouds generated using structure from motion systems have become prevalent in many domains and industries. Technology such as AR/VR have made heavy use of SfM and are used in locations that range from Military installations to private homes. It is also being deeply integrated our lives as many phones are now able to generate such point clouds using the same technology. There is however a glaring problem of preventing scene recreation attacks. These attacks will be able to reveal private and confidential information from secret facilities as well as private homes. We propose a novel solution where we add noise into the point cloud to prevent such attack. This added noise should not reduce the localization accuracy of the point cloud but will prevent scene recreation attacks from occurring. While previous papers have used adversarial attacks to fool classification networks into misclassifying point clouds, There is no currently known research in using this method of attack to prevent scene recreation attacks. To add noise we will be utilizing diffusion generative models. s. We will train this diffusion model to first add random gaussian noise and then try to interpolate the noise back to the original point cloud while maximizing localization metrics and minimizing scene recreation attack vulnerability. This optimization will be achieved by modifying the loss function being used. The two primary questions are can we add noise to prevent scene recreation attacks and can we add it without harming the overall accuracy of the model. We managed to prove that adding noise can prevent such attacks but we’re not able to finish our diffusion model.
Audio Transcript
– Hello everyone and today i wanted to go over my research project for this semester.
– My recent project was titled securing SFM Systems using Generative Modeling however i just like to call it you’re missing the point.
– So before we talk about the bigger questions that this project aims to answer let’s talk a bit more about the underlying concepts.
– One of them is point clouds so what is a point cloud? A point cloud is just a way to represent a 3D structure
– As you can see here this is a 3D representation of a building, and they’re generated by taking a bunch of 2D images, usually at different angles and then trying to match up the different points in those images and projecting them into a 3D space, and that is what a point cloud is, and they’re used in many technologies. Technology such as self-driving cars where point clouds are used to help the car understand its environment and where it is in relation to other objects. In AR/VR systems it’s used to understand where you are in relation to your environment, let’s say for example your living room and these systems are becoming more and more widely adopted.
– While a lot of time and energy has been spent into making these systems more efficient as well as more available for a broader use, not as much effort has been spent in making these systems more secure. Specifically secure against a type of attack known as scene recreation attacks. This is a type of attack where you can take the original point cloud which is just supposed to have structural information and then recreate the images that were used to generate that point cloud. While in some contexts this attack doesn’t lead to any loss of meaningful information in other contexts such as our home it can be a huge invasion of privacy, also as these systems are being used in more secure installations such as military bases it can also constitute a security threat.
– So after understanding these concepts we can begin to take a closer look at the questions we’re trying to answer in this research project. So the first one is, can we prevent these attacks by adding random noise around the existing point cloud and secondly can we do it or can we add noise in such a way that it doesn’t harm the overall accuracy of this model.
– So to answer the first question we decided to create a proof of concept demonstration where we took an existing point cloud and added lots of random noise all around it and as you can see here by adding this noise we were able to prevent a scene recreation attack from occurring however because these points were added randomly we also lost a lot of the accuracy of this model.
– So to preserve this accuracy we are going to use deep learning, specifically we’re going to use deep learning to understand how this added noise will affect the actual accuracy of the model as well as its susceptibility to the scene recreation attack and then we’re going to use this relationship to optimize the added noise.
– specifically we’re going to be using a type of model called diffusion modeling. This is a type of model where we take the original point cloud and then keep on adding noise onto it until it’s just random noise and then we train our network to learn how to take this random noise and recreate the original point cloud from it however we slightly modified it where instead of just learning how to recreate the original image it will also learn how to do it in a way where we decrease our susceptibility to the scene recreation attack. While this is an incredibly powerful model it is also very complex and it takes a long time to not just train but to sample from as well and as a result we were not able to fully finish this model by the time of this recording.
– therefore I would like to continue developing this model further in the future as well as making this model available to other developers for feedback as was integration with their existing pipelines and finally i would also like to look for other methods that might yield similar results while reducing the time it takes for the development.
Thank you all so much for your time and i would like to take a second to thank professor Bo Han for being my mentor and guiding me through this process Mr Nan Wu for helping me answer a lot of the questions I had while I was in development ,the GMU OSCAR office for teaching me about research as well as providing funding for this entire project, the ORC office here at gmu for providing computational resources and all others who have made this possible including my classmates, my friends as well as my family thank you so much

Leave a Reply