Security Now
Security is top-of-mind everywhere, especially right here where Juniper experts share their thoughts on the latest security breakthroughs and product advancements
Showing results for 
Search instead for 
Do you mean 

Return-oriented programming and malware detection

by Juniper Employee on ‎10-04-2016 08:00 AM

Return-oriented programming, or ROP, is a clever technique used to get around the NX (No-eXecute) and DEP (Data Execution Prevention) mitigations in modern CPUs and operating systems. Traditionally, exploiting a vulnerable program has consisted of the following:


  1. Find a programming error in the application (for example, Adobe Flash) that allows specially crafted input to overflow or otherwise corrupt an allocated region in memory.
  2. Inject executable code (shellcode) into the program’s memory
  3. Transfer control to this new code by overwriting control information such as a return address on the call stack. 

With this technique, simply opening a document or media file results in arbitrary code being executed on the target’s system. The attacker has control and can download additional malware, exfiltrate information, install spyware or hooks for persistence, etc.


Here's a simple example of a program vulnerable to a memory corruption attack. The attack takes advantage of the insecure gets() function, which reads a string from the console and writes it to a specified location in memory without checking whether there is enough space available.




When the function buggy() is called, the computer stores the return address (the next instruction in main() following the function call) on the program's call stack in memory. After buggy() finishes, the program execution should return to the address 0x00401047 (seen reversed here in the program's memory because x86 is a little-Endian architecture).



The name entered by the user is also stored on the stack in the 8 characters allocated for 'str'.


But if we enter more than 8 characters, the gets() function will happily overwrite adjacent memory:



Note that the final two characters -- the final 'n' in Langton and the null character used to indicate the end of the string -- have clobbered half of the return address. The result is a crash as the control jumps to the garbled address 0x0040006E. But an attacker can go further by including executable shell code in the input string and overwriting the original return address so that program control now jumps to the attacker's own code.


NX (No-eXecute) and DEP (Data Execution Prevention) counter this type of exploit by ensuring at the hardware level that a given section of memory is writable or executable, but not both. Even if the attacker finds a memory vulnerability and injects shellcode, the CPU will refuse to execute those instructions. Return-oriented programming gets past this limitation by (mis)using the existing code in the application in ways that were not intended.


To understand how this works, we’ll start with an analogy. In Wisconsin, there is an executive power known as the Frankenstein veto, which allows a governor to selectively reject individual words of a proposed bill. Here’s an example:



By selectively vetoing single words and phrases, the proposed law was changed from:


[...] the secretary of administration shall lapse to the general fund or transfer to the general fund from the unencumbered balances of the appropriations to state agencies, as defined in subsection (1w) (a), other than sum sufficient appropriations and appropriations of federal revenues, an amount equal to $724,900 during the 2006−07 fiscal year [...]




[...], the secretary of administration shall transfer from the balances of the general fund  an amount equal to $330,000,000 during the 2005−06 fiscal year and the 2006−07 fiscal year [...]


By selectively repurposing the existing text, the governor changed an appropriation by nearly three orders of magnitude. (An earlier incarnation of this executive power allowed governors to “veto” individual letters in a bill!)


Similarly, return-oriented programming reuses existing snippets of the vulnerable application’s instructions for purposes that were not intended. By overwriting portions of the call stack, the ROP exploit jumps around the program, each time selectively executing a small number of instructions preceding a function’s 'ret' statement (hence the name).


As we saw above, we can exploit a programming error to overwrite a function's return address on the stack. This allows us to transfer program control to an arbitrary location. Consider the following function fragment:



By setting the return address to 0x0040A4BB while overwriting the stack, we jump into the end of this function, setting the register eax to 0 by XORing it with itself. The return instruction at 0x0040ABD expects to find another return address on the stack, but this too can be overwritten along with the previous address. A chunk of code (mis)used in this fashion is called a ROP gadget, and a ROP exploit is formed by a chain of such gadgets called in sequence due to the intentionally corrupted stack memory. Due to the difficulty of constructing a suitable sequence of bytes to overwrite the stack and control program flow through a sequence of function fragments, this technique is used only as long as necessary; a common approach is to use ROP to bypass DEP, and after that to use more traditional techniques to complete the exploit.


With an understanding of the mechanism behind ROP exploits, we get to the question that prompted this post: does Sky ATP attempt to directly detect ROP exploits? The answer is no, for the following reasons:


  1. ROP detection is redundant in an anti-malware sandbox. ROP is only used as a stepping stone to run malicious code on a device. The purpose of malware is to do something malicious: ransomware, a backdoor to add the victim's computer to a botnet, data exfiltration, etc. Sky ATP's dynamic analysis engine detects a rich set of malicious indicators, whether or not the malware's initial foothold came from a ROP exploit.
  2. ROP exploits are system-specific and fragile, so detecting them in a sandbox is highly unlikely. As we saw previously, ROP exploits target programming errors in an installed application, and so the sandbox must also be configured with the same vulnerable version. Beyond this, since ROP exploits jump around raw executable code, small changes to a sandbox's configuration (different versions of system libraries, hardware variations, etc.) frequently render an attack as inert, resulting in -- at most -- an application crash, not a successful exploit. One should ask anti-malware vendors touting their ROP detection how many times they have detected an actual ROP exploit in the wild.
  3. ROP detection is very resource-intensive. Monitoring system activities at the hardware level needed to observe ROP patterns in the program flow is both very slow and much easier for evasive malware to detect.

The low probability of actually detecting a live ROP exploit must be balanced against the high cost of CPU-level emulation. Multiply this by the number of systems a sample must be run on to ensure some likelihood of an exploit/vulnerability match, and the cost-benefit ratio is abysmal at best. And because a successful exploit would also be detected by behavioral indicators, most anti-malware solutions -- including Sky ATP -- do not perform direct ROP detection.

by Distinguished Expert
on ‎10-04-2016 08:38 AM



Very interesting post. Do you mean that ROP is not checked by SkyATP, but regular code injection is still detected? Using similar argumentation, I could come to the conclusion that shellcode injection does not need to be detected as well, being an intermediate step in malware's functionality.


by Juniper Employee
on ‎10-04-2016 10:11 AM

To clarify, there are two types of code injection common with malware:

  1. Malware injects code into another process to hide its activities. This is a common technique, and we detect many forms of this. It's a good indicator of malicious intent.
  2. Specially-crafted input to a vulnerable application results in memory corruption and arbitrary code execution. This is extremely difficult to detect in general, hence the mitigations: DEP, NX, stack guards and canaries, runtime bounds checking, etc. Rather than use expensive and unreliable heuristics to try to detect such an exploit, Sky ATP detects unusual behavior by the application itself, and -- because our verdicts are based on machine-learning -- we're able to leverage any unusual system activities for this purpose.
by Distinguished Expert
on ‎10-04-2016 12:44 PM

Thanks Asher, appreciate your clarification.

Juniper Networks Technical Books
About the Author
  • Andrew is a Juniper Distinguished Engineer responsible for the architecture of Juniper's network management user interfaces.
  • Amy James is Product Marketing lead for Security at Juniper Networks. She brings her knowledge of cyber security from companies like FireEye, Cisco and Cloudmark with deep roots in technology storytelling.
  • Asher Langton is a senior software engineer and malware researcher on Juniper's Sky ATP team.
  • Aviram Zrahia is a consulting engineer at Juniper Networks and an industry researcher of cyberspace. He holds a CISSP and GCIH certifications, as well as a bachelor's degree in computer science and MBA in management of technology, innovation, and entrepreneurship. He is also a research fellow in the Blavatnik Interdisciplinary Cyber Research Center (ICRC) at Tel Aviv University, currently focusing on the domain of threat intelligence sharing.
  • Brad Minnis, CPP is the Senior Director of Corporate Environmental, Health, Safety & Security for Juniper Networks, Inc. based in Sunnyvale, CA, where he is responsible for strategic design, implementation and management of the company’s security, safety, environment, crisis management and business continuity functions. He also leads the company’s efforts in corporate citizenship and sustainability, and manages the Corporation’s government-related security programs. Mr. Minnis has over 30 years experience in the Silicon Valley and has managed EHSS operations for a number of high tech companies, including Juniper Networks, 3Com Corporation, and National Semiconductor Corporation. Mr. Minnis’ specialties include security management, supply chain and product integrity, anti-counterfeit, occupational health and safety and crisis management. In his role as Cyber Incident Response Team Leader for Juniper, Mr. Minnis has managed numerous high impact cyber-related incidents and cross-functional responses. Mr. Minnis served for ten years in the United States Navy and has served in leadership positions the International Security Management Association (ISMA) and ASIS International, serving as Chairman of the San Francisco Chapter in 2003. He has also co-written several publications on software integrity assurance and supply chain security with organizations such as SAFECode. Mr. Minnis is certified as a Protection Professional by the Professional Certification Board of ASIS International and attended the University of Connecticut, where he received two certificates in Environmental, Health and Safety
  • Bill is the Director of Federal Certifications and Policy at Juniper Networks. In this role, Bill focuses on several areas unique to the needs of Federal Government customers, including product certifications, IPv6, and security. Bill came to Juniper Networks in January 2008 after more than 20 years in the IT community working with commercial enterprise customers, service providers, and the US Federal Government. Bill started his career as an engineering officer in the US Air Force after graduating with a Bachelor of Aerospace Engineering from the Georgia Institute of Technology. Bill has an MBA from the Wharton School at the University of Pennsylvania.
  • Craig Dods is the Chief Architect for Security within Juniper Networks' Strategic Verticals. He currently maintains multiple top-level industry certifications including his JNCIE-SEC, holds multiple networking and security-related patents, as well as having disclosed multiple critical-level CVE's in a responsible manner. Prior to joining Juniper, Craig served as IBM's Managed Security Services' Chief Security Architect, and held previous security roles at Check Point Software Technologies and Nokia.
  • François Prowse is a Senior Systems Engineer for Juniper Networks, based in Brisbane Australia. Francois joined Juniper in 2006 as part of the New Zealand SE team, subsequently relocating to Australia. Prior to Juniper, Francois worked for four years at Alcatel in both operational and architectural roles, being jointly responsible for the construction of New Zealands' largest MPLS core network. Prior to Alcatel, Francois worked at UUnet, focusing on core network expansion in Europe. In all previous roles JUNOS has been the driving factor behind day to day operations, providing him with over 8 years of operational experience. Francois is a Juniper Networks Certified Internet Expert (JNCIE #144) which he obtained prior to joining Juniper Networks.
  • Greg Sidebottom is a Senior Engineering Manager in the Identity and Policy Management business unit at Juniper Networks. Greg has spent the last decade plus conceptualizing, architecting, designing, and leading the implementation of Juniper's SDX and SRC families of policy based service management applications. Previous to this, Greg held positions in the software and networking industries at Siemens, Cognos, Nortel, GTE labs subsidiary MPR Teltech, and the Alberta Research Council. Greg is an author of eight invention disclosures resulting in two patents issued and three pending. Greg holds a B.Sc. in Computer Science for the University of Calgary and an M.Sc. and Ph.D. in Computing Science from Simon Fraser University.
  • Jennifer Blatnik is vice president of cloud, security and enterprise portfolio marketing at Juniper Networks with focus on enterprise deployments of security, routing, switching, and SDN products, as well as cloud solutions. She has more than 20 years of experience helping enterprises solve network security challenges. Before joining Juniper, Jennifer served multiple roles at Cisco Systems, Inc., including directing product management for security technologies aimed at small to medium enterprises, as well as supporting managed services, cloud service architectures and go-to-market strategies. She holds a B.A. in Computer Science from University of California, Berkeley.
  • Jim Kelly, Senior Product Line Manager – CTP Products Juniper Networks. Jim Kelly is the senior product line manager for the CTP products where he is responsible for the CTP product direction, marketing and circuit emulation applications within Juniper Networks. Mr. Kelly has more than 28 years of experience in the networking industry in technical roles, sales, marketing, and product management positions. He started his career in the United States Air Force. He has worked for Wang, Digital Telecom Systems, American Airlines, Network Equipment Technologies, Carrier Access, and Nortel Networks. He started Juniper Networks federal DoD sales in July 2000 and joined Juniper Networks again in October 2005 through the acquisition of Acorn Packet Solutions where he was the director of sales and marketing.
  • I have been in the networking industry for over 35 years: PBXs, SNA, Muxes, ATM, routers, switches, optical - I've seen it all. Twelve years in the US, over 25 in Europe, at companies like AT&T, IBM, Bay Networks, Nortel Networks and Dimension Data. Since 2007 I have been at Juniper, focusing on solutions and services: solving business problems via products and projects. Our market is characterized by amazing technological innovations, but technology is no use if you cannot get it to work and keep it working. That is why services are so exciting: this is where the technology moves out of the glossy brochures and into the real world! Follow me on Twitter: @JoeAtJuniper For more about me, go to my LinkedIn profile:
  • Kevin Walker is the Security Chief Technology and Strategy Officer for Juniper’s Development and Innovation (JDI) organization. He is responsible for driving the security strategy both internally within Juniper, and externally with investors, partners, influencers, and customers. He provides the guidance required for JDI to conceive, develop and create momentum for industry-leading security solutions. Working closely with the Security Engineering team, Walker identifies the opportunities for improved security, growth, and innovation to deliver the scalable, reliable, and compliant security architecture needed in today’s security landscape. Before joining Juniper, Walker was VP and Assistant Chief Information Security Officer (CISO) at He has served as a Chief Information Security Officer (CISO), Chief Security Strategist and Director of Information Security across a number of notable companies including Intuit, Cisco, Symantec and VERITAS Software. With over twenty-five years in various computer science and information technology disciplines, focusing on enterprise applications, network design, and information security, Walker possesses research and engineering expertise across of range of technologies including networking protocols, securing applications at the atomic level, cryptography, and speech biometrics.
  • Security Life timer, who has been described as a true IT security ‘guru’. It is certainly apt: his knowledge and expertise developed over the course of more than 20 years in IT have helped many customers implement a security strategy that not only safeguards their business and information, but enables Digital Transformation. A noted public speaker on security issues, Lee’s passion and style stand out in the sometimes staid world of network security. Prior to joining Juniper Networks, Lee held a number of business and technical roles at Dr Solomon’s, McAfee, Hewlett Packard, Nokia Siemens Networks and Citrix. Lee leads the Juniper Networks security business across Europe, Middle East and Africa. In this role, Lee is responsible for the company’s commercial development in the field.
  • Laurence is passionate about technology, particularly cyber security. His depth and breadth of knowledge of the dynamic security landscape is a result of over twenty years’ experience in cyber security. He understands the security concerns businesses face today and can bring insight to the challenges they will face tomorrow. Laurence joined Juniper Networks in 2016 and is our senior security specialist in EMEA. Security throughout the network is a key area where Juniper Networks can help as business moves to the cloud and undertakes the challenge of digital transformation.
  • 30 Years in Book Publishing, 20 years in Technical Book Publishing, including Apple Developer Press, Adobe Press, Nokia Developer Books, Palm Books, and since 2001, almost 10 years as consulting editor/editor in chief for Juniper Networks Book. Joined the company and started the Day One book line and in 2011, the new This Week book line.
  • Paul Obsitnik is Vice President of Service Provider Marketing for Juniper Networks Platform Systems Division (PSD), responsible for the marketing of Juniper’s portfolio of high performance routing, switching, and data center fabric products to Service Providers globally. Paul's team is responsible for marketing strategy, product marketing, go-to-market planning, and competitive analysis worldwide for the Service Provider segment. Obsitnik has extensive experience in marketing, sales and business development positions with a proven track record in creating technology markets. He has served in senior marketing and sales management positions at several companies including BridgeWave Communications, ONI Systems, NorthPoint Communications and 3Com. Paul holds a Bachelor of Science with Honors in Electrical Engineering from the United States Naval Academy and a Master of Business Administration from the Harvard Graduate School of Business. Obsitnik is based in Sunnyvale, California.
  • Solutions Marketing Sr Manager
  • Michel Tepper is a Juniper consultant and instructor working for Westcon Security in the Netherlands. He started working in ICT in 1987. Michel is also is a Juniper Ambassador. Currently he holds three Junos Professional certifications and a number of specialist and associate certifications on non-Junos tracks. Michel is an active member of J-Net and, where he uses the nickname screenie referring to the ScreenOS with which he started his Juniper Journey.
  • Scott is the Director of Product Marketing for Mobile Security at Juniper Networks. In his 20+ years in high tech, Scott has worked on Mobile and Endpoint Security, Network Security, IPS, Managed Services, Network Infrastructure, Co-location, Microprocessor Architecture, Unix Servers and Network Adapters. He has held leadership roles at Check Point, McAfee, Symantec, Exodus Communications, Cable & Wireless, Savvis, and HP.
  • Sherry Ryan is IT Vice President and CISO of Juniper Networks. Previously, Sherry held similar positions at Blue Shield of California, Hewlett-Packard, Safeway and Levi Strauss where she established and led their information security programs. Sherry holds the Certified Information Security Manager (CISM) certification from ISACA and the Certified Information Systems Security Professional (CISSP) certification from ISC2. She is a member of the High Tech Crime Investigation Association (HTCIA) and the Information Systems Security Association (ISSA). Sherry has a bachelor's degree in Business Administration from the University of Redlands, and earned her MBA from the College of Notre Dame.
About Security Now

Discussing a wide range of topics impacting enterprises and
data center security.

Subscribe RSS Icon

Our Bloggers

Kevin Walker
Vice President
Security CTSO, Engineering

Profile | Subscribe

Ritesh Agrawal
Software Engineering

Profile | Subscribe

Scott Emo
Product Marketing

Profile | Subscribe

Bill Shelton
Director Field Sales

Profile | Subscribe