This is an interesting case that I recently worked on and would like to share as part of this series. An organization that uses Exchange Online Protection was receiving automated emails from a partner. These emails were arriving with empty subjects and bodies. When the partner sent the automated emails to a Hotmail.com / outlook.com address, they would also arrive with no subject or body.

 
However, when the partner sent these emails to a Gmail account, the would arrive just fine with the subject and body intact.

 
Huh. So what’s going on here?

Problem and Environment

If it wasn’t for the successful Gmail test, I would have immediately said the problem was with the third party sending application / server. Test messages from the third party looked fine in Gmail, but in Hotmail.com / Outlook.com, and after passing through EOP, the messages showed no subject or body.
This organization had their mailboxes in an on-premises mail environment that was protected with Exchange Online Protection.

Troubleshooting

Test 1 – Reproduce the problem

First things first, I first want to reproduce the problem. Maybe I will get lucky and everything will now be working! So I have the third party send a single email from their application server to the following recipients.

• My customer
• My gmail.com account

The customers on-premises mailbox received the message and indeed there was no subject or body. I received the same message in my Gmail account and saw subject and body for the message. I verified that the copy my customer received traversed through EOP before being delivered to their on-premises environment.

Ok, so the problem is still occurring and incredibly easy to reproduce. Let’s move on.

Test 2 – Have the third party send to more accounts

Now I’m curious if the problem is related only to this customer and their EOP tenant, or will the same thing happen for any organization using Exchange Online or Exchange Online protection? Next I ask for the third party to send another test message to a range of accounts. Following are the recipients accounts along with the results.

• My work account (hosted in Exchange Online) – message arrived with no subject or body
• A second Exchange Online mailbox – message arrived with no subject or body
• My Outlook.com account – message arrived with no subject or body
• My gmail.com account – message arrived with the subject and body intact

These results surprised me. I did not anticipate Outlook.com also not being able to display the subject or body.

Based on experience, I know that some mail services follow the SMTP RFC specification much more strictly than others. I also know that Microsoft follows the SMTP RFC quite strictly.

My guess at this point is that the third party is sending the message via SMTP that is not within the SMTP RFC specification. And so when a Microsoft email service (consumer or business) receives the message, it cannot be parsed properly and thus we end up with the subject and body missing.

Test 3 – When in doubt, run WireShark

A network trace never lies, and so next I turn to WireShark to either prove, or disprove my guess that the third party is sending outside the SMTP specification. I setup WireShark on my Exchange 2013 server and have the third party send me a message directly. Here’s what I captured.

The connection to my server was made fine, however when it was time for the DATA to be sent, WireShark interprets the received packet as being Malformed. While this doesn’t always represent a problem, it certainly requires investigation.

In the same capture, I received a message from my customer in which I can see the IMF packet was received fine and is interpretable by WireShark (not malformed), so I know that WireShark doesn’t just interpret all IMF packets as malformed.

Test 4 – Telnet test

I know that the third party is using an application to send these messages. Next I’d like to try and isolate the sending server from the sending application. I ask the third party to send me a message from their application server, but using Telnet.

I received the message just fine in both my Outlook.com account and my Exchange Online account, both subject and body were intact. Based on this troubleshooting, the problem is looking to be with the partners sending application. My guess still is that the application is not sending within the SMTP RFC, and that GMail is more lenient to the spec which is why it will show the subject and body of the messages sent from the application.

Test 5 – Final test

As a final test, I asked the partner to run a network capture on their application server and capture the application sending an outbound message. This network trace also showed an unreadable packet coming out of the application server. At this point I’m feeling pretty confident that the problem is with the sending application.

Resolution and Root Cause

After investigation, the partner found that their application was not properly inserting Carriage-Return Line-Feeds into sent messages. This caused the sent messages to be out of spec for the SMTP RFC and caused Outlook.com and Exchange Online to not properly parse the message.

The partner updated their code so that it properly inserted line-feeds, and sent another round of test messages. This time the messages showed up just fine in Outlook.com and Exchange Online, showing both subject and body.

Success!!

Wrap-up

In a problem like this there is no right or wrong order of what to try and where to look. My approach is just one of many that could have been taken. In the end, my goal was to systematically rule out the various points in the mail flow to get down to the component that was causing the problem. Mission accomplished!

Every week I work with multiple customers that have experienced phishing attacks where their own domain has been spoofed by the attacker. The conversation always revolves around implementing SPF (Sender Policy Framework) and DMARC (Domain-based Message Authentication, Reporting, and Conformance) to secure their domain. For organizations that cannot use SPF because of its limits (the record is restricted to 10 DNS lookups), we usually discuss DKIM (DomainKeys Identified Mail).

You (yes you!) can currently enable DKIM on your Office 365 tenant through a manual process. My colleague, Terry Zink, has a great blog post on the steps to take to enable DKIM signing for outbound mail from your Office 365 tenant: Manually hooking up DKIM signing in Office 365. Please check out his article (hyperlink in the previous sentence) for the steps on enabling DKIM.

If you have out grown the limitations imposed by SPF, then it’s time to investigate implementing DKIM.

Why DKIM and not just SPF?

The biggest reason companies turn to DKIM is because of a limitation on SPF. SPF records can only contain ten DNS lookups. If you are an organization that uses a lot of third party companies to send mail on your behalf (where they spoof your domain) then your SPF record may contain more than ten DNS lookups, which essentially renders it useless for protecting your domain against spoofing. Most receiving servers will stop evaluating SPF records once they have hit the DNS lookup limit of ten and at that point will stamp “permerror” for the SPF result.

This is where DKIM comes in. First let’s look how it works, and then we’ll look at how it overcomes the DNS lookup limit imposed by SPF. From an EXTREMELY high level view, here is how DKIM works.

Prep Work

• You obtain a certificate for DKIM signing and publish the public key in your public DNS.

Outbound Mail

• Your mail server calculates a hash for all outbound messages.
• This hash is then encrypted with the private key of your DKIM certificate, and this value is placed in the header of the message

Receiving Server

• The Receiving server obtains the sending domains public DKIM key from DNS and uses this to decrypt the hash value that the sender placed in the header. We’ll call this value X.
• The receiving server then re-calculates the hash for the inbound message. We’ll call this value Y.
• If X = Y, the receiving server knows that the message was not tampered with in transit.

As a sender, you can give your private key out to any number of third parties that are allowed to send on your behalf. There is no limit, and this is how DKIM overcomes the DNS lookup limit imposed by SPF.

Even with DKIM, you will still want to publish a DMARC record to prevent attackers that spoof using different MailFrom and From headers. See my previous article Using DMARC to Prevent Spoofing.

For DMARC to pass, you need either SPF or DKIM to pass. This means that you can leave an existing SPF record in place while you implement DKIM. Third parties that don’t support DKIM can remain in your SPF record, while third parties that do support DKIM only need to be giving your DKIM private key.

Wrap up

The above explanation of DKIM is from 60,000 feet. It is very high level and there are many configuration options and decisions that you’ll need to consider before implementing DKIM. For technical specs on DKIM, DMARC, and SPF, see the links in the Resources section of the article.

Resources

Manually hooking up dkim signing in Office 365
How Office 365 does automatic DKIM key rotation
Using DMARC to prevent spoofing
DKIM
SPF
DMARC

Transport rules contain an Audit setting that is often misunderstood and unchecked without realizing the implications.

Unchecking this box has quite adverse effects on future reporting and troubleshooting for the transport rule. While this may be desirable, I see a lot of organizations unchecking this box and not realizing what the impact will actually be.

By default, newly created transport rules have the Audit this rule with severity level box checked, and the severity level is set to Not specified. When looking at this option, two questions come to mind.

1. What does the checkbox actually do?
2. What do the various audit severity levels actually do?

I’m going to break the above two questions into their own sections.

What does the Audit checkbox actually do?

Now this is really important, so I’ll put this on its very own line.

The checkbox controls whether this rule will appear in reports and message traces.

If you uncheck this box, there will be no trace of this rule triggering anywhere. If the rule has triggered, you will see no evidence of this in reports or the message trace, and not even in the extended message trace. This obviously will have a huge impact on troubleshooting if you are looking to see if this rule has triggered or not.

Example

Consider the following transport rule.

If a message is sent from an external sender, this rule will prepend text in front of the subject. This rule also has Auditing disabled. Let’s take a look at a message trace for a message that triggered this rule.

We can see that the rule triggered based on the modified subject. However, there is no trace of the rule triggering in the details of the message trace.

Similarly, when looking at an extended message trace, we will also see no evidence of the rule triggering. The following was extracted from an extended message trace for this message.

Note: A triggered transport rule will show up in the extended message trace under the custom_data column as S:TRA=ETR|ruleID…, and this is not present in the above screenshot.

Finally, and this won’t come as a surprise, if we view a report of the above rule it will not show when or if it has triggered.

The reason for this behavior comes from a privacy perspective. It is meant to be used by organizations that have mandates in place that prevent tracking this type of information.

This behavior is noted in the TechNet article Transport rule actions.

If you clear the Audit this rule with severity level checkbox, rule matches will not show up in the rule reports.

To wrap this section up, unless you need to disable auditing, I would recommend not unchecking this box. It’s incredibly hard to troubleshoot something that does not appear in any logs.

Audit severity

The second item we need to investigate is the severity level that’s associated with the audit option. By default, a new rule will have this set to “not specified,” but you can change it to the following options.

These levels impact how the rule will show up in reports, that’s it. When viewing reports, you can filter based on audit level. If you only want a report to display triggered rules that have an audit level set to either medium or high, you can set that filter like this.

Rules that have an audit level set of “not specified” will show up in reports as having a “low” severity set. Regardless of the set severity, as long as the Audit check box is checked, there will be logs of the rule triggering (message trace, extended message trace, reports).

Summary

Unless you are mandated to not record data on transport rules triggering, I’d recommend not unchecking the Audit checkbox on any transport rule. Unchecking this box will make troubleshooting that involves transport rules very difficult.

Regular message traces are sufficient for most mail flow troubleshooting, but occasionally we need more data which requires obtaining an extended message trace. These traces (provided as a CSV file) contain a plethora of information, however parsing them can be a very overwhelming experience.

I would like to share the work flow that I use when I need to extract data from an extended message trace. Hopefully this will help make your own experience a more pleasant one.

Obtaining an extended message trace

Obtaining an extended message trace can be tricky if you’ve never requested one before. In the Message Trace tool, set a custom date range and ensure the starting date is at least a week in the past (this is the key to enable the extended trace). Add either a sender, recipient, or Message ID, and check off the box Include routing events and routing details.

These traces typically take at least an hour to complete. Once completed, you can download the CSV file at the link View pending or completed traces on the Message Trace page.

For more information on running a message trace see Run a Message Trace and View Results.

When opening the resulting CSV with Excel, you’ll see something like this.

Yikes right? Well, with a little Excel magic, we can make this file much more workable.

Parsing an extended message trace in Excel

After opening the CSV file in Excel you should see something like the previous screenshot. Excel should split the data up into columns automatically, but if not you’ll need to perform a manual import and select a comma as the delimiter.

Sort the data chronologically

Message traces do not come sorted by date and time (in fact they aren’t sorted by anything), and so this is the first order of business.

1. Highlight the first row, then click the Data tab, and then Filter.
   

2. To make reading our headers easier, highlight the entire first row again, and then click Format under the Home tab, and then Auto Fit Column Width.
   

3. Expand the date_time column (first column) so you can see the full time stamp, then click the filter drop down for that column and select Sort A to Z.
   
   

Now that we have our trace sorted chronologically, we next need to start removing messages that aren’t part of our investigation.

Remove High Availability entries

High Availability entries can be ignored 99% of the time. These can be identified by entries under the event_id column which start with HA. To remove these rows, just filter them out through this column.

The exception here is HASUBMIT. If you see this listed, do not filter it out.

Only show rows pertaining to a particular message

Typically, we only want to see details from a particular message. The easiest way to do this is to filter based on the Message ID. If you don’t have the Message ID, you can also filter based on Sender, Recipient, Subject, etc. For this example, I’m going to filter based on Message ID.

Find the message_id column, click the filter drop down, and then uncheck select all. Now find your Message ID in the list, check it off, then hit OK.

I am now looking at 7 rows, compared to my starting point of over 7,000 rows! Each of the now remaining rows represent a different EOP server that this message traversed through.

Some rows are more interesting than others, and typically the most important row is where the EOP header (x-forefront-antispam-report) values are stamped on the message.

Extracting the EOP (x-forefront-antispam-report) header

One of the most important items we can pull from the extended message trace is the x-forefront-antispam-report header information for a message. This is especially helpful when we do not have the original .msg file for a message we are investigating.

This data can be found in the custom_data column. Look for the cell that starts with S:AMA=.

For my workflow, I copy the contents of this cell and paste it into Notepad ++ which results in this view.

To clean this up, I use the Replace action. Use the following options in the Replace window, and then click Replace All.

I can now very easily pick out the various EOP results.

Some interesting items we can quickly pull from here as follows:

• S:AMA – Anti Malware Agent: Will indicate if any viruses were detected.
• S:TRA=ETR – Transport Rule Agent: ETR|ruleid=[guid] will indicates the GUID of any transport rules that triggered (Note: S:TRA=ETRI and S:TRA=ETRP do not indicate rules that have been triggered)
• S:SFA = Spam Filter Agent: Will display contents from the X-Forefront-Antispam-Report header (Spam Filter Score, Spam Filter Verdict, message direction, etc…)

Here is the S:SFA string from an extended trace with the SCL, SFV, and direction values highlighted.

S:SFA=SUM|SFV=SKN|IPV=|SRV=|SFS=|SCL=-1| SCORE=0|LIST=0|DI=|RD=|H=BLUPR06MB849.namprd06.prod.outlook.com|CIP=|SFP=0|ASF=0|HCTFP=|CTRY=|CLTCTRY=US|LANG=en|LAT=0|LAT=0|LAT=0|FPR=|DIR=INB

This was an inbound message which EOP gave an SCL of -1 to. The Spam Filter Verdict for this message is SKN which indicates the safe listing was done prior to the spam filter (ex. Transport rule).

For more information on the custom_data column, see Run a Message Trace and View Results. On this page, expand information under the “View message trace results for messages that are greater than 7 days old.”

EOP Outbound Connector that was used

For mail that is leaving your Office 365 tenant, the extended trace can tell us which EOP outbound connector was used to deliver the message. For an outbound message, look in the cutom_data column for a row that begins with S:E2ELatency=. In this example, my outbound message was sent using an EOP outbound connector that I created called xbox.com outbound.

S:E2ELatency=32.140;
S:ExternalSendLatency=3.155;
'S:Microsoft.Exchange.Hygiene.TenantOutboundConnectorCustomData= Name=xbox.com outbound; 
ConnectorType=Partner; 
UseMxRecord=False'; 
S:OutboundProxyTargetIPAddress=xx.xx.xx.xx;
S:OutboundProxyTargetHostName=xx.xx.xx.xx;…

I can also see that this connector is type Partner, and uses a smart host for delivery because UseMXRecord is set to false. The OutboundProxyTargetIPAddress property shows us the recipient IP (I have obfuscated the actual IP in the text above).

Server SMTP responses

Another interesting piece of data we can pull from extended traces are the SMTP server responses. Earlier this year, this information was made available in standard message traces so you no longer need to run an extended trace to obtain it.

If you do have an extended trace, server responses can be seen under the recipient_status column. Here’s an example of Office 365 being unable to deliver a message. The recipient server responded to the delivery attempt with a 450 error stating a certificate problem.

'450 4.7.0 Proxy session setup failed on Frontend with ''451 4.4.0 Primary target IP address responded with: "454 4.7.5 Certificate validation failure." Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts, or delivery failed to all alternate hosts. The last endpoint attempted was xx.xx.xx.xx:25'''

Delivery folder

For inbound messages destined to mailboxes hosted in Exchange Online, an extended message trace will show us which folder in the recipient’s mailbox the message was delivered to.

Note: This only works if the recipient mailbox is located in Exchange Online, and only works for server side mailbox rules that cause a message to be delivered to a particular folder.

For example, here’s an inbox rule that I have configured on my Exchange Online mailbox.

When I run an extended message trace on a message that would have triggered this mailbox rule, I can see the folder where this message was delivered.

Just look for the DELIVER event in the event_id column, and look at the recipient_status column. A semi-colon in the recipient_status column indicates that the message was delivered to the recipient’s inbox.

Summary

I’ve only just scratched the surface of what data can be extracted from extended message traces. While these traces can initially be overwhelming, the above tips, combined with experience will make extracting data from them much easier.

Resources

Run a Message Trace and View Results
Message Trace FAQ

I recently came across an attack on my Exchange 2013 lab server and want to share I saw. While this post isn’t directly related to EOP, it is certainly security related and a good reminder of what’s out there.

I have an Exchange 2013 lab server running in Microsoft Azure and was recently performing network captures on it. At the end of my work day I had a network capture running that was scoped to port 25 and forgot it running when I left the office.

In the evening I logged back in to the server to stop the network trace and was very surprised to see the network trace log scrolling very quickly with new traffic on port 25. The only traffic this server typically sees is from my own testing.

Just before 20:00 my server was connected to on port 25 from an IP that identifies itself with EHLO scanner.sslsonar.org (this seems to be a bogus domain). In this screenshot 10.0.0.5 is the private IP of my lab server.

It reaches out to my server on port 25 with an EHLO and my server responds with an SMTP Hello since I have not configured any restrictions on its Receive Connector. The above IP reverses to Amazon Web Services. It looks like someone has a VM in AWS that is running a port scanner.

Twenty-five minutes after the above connection, I start getting flooded with authentication attempts from a different IP (this IP resolves to the countries Antigua and Barbuda, however it could just be a proxy and not the actual origination of the attack). I currently have TLS turned off on my Receive Connector so all the data is clear in the network trace.

Here’s what I see in the trace :

1. Attacker connects and starts a session with EHLO 127.0.0.1.
2. My server responds and the attacker responds with AUTH LOGIN.
3. Attacker then uses various user names and passwords which are encoded in Base64. Decoding the usernames provide names of accounts that are typically default on non-Exchange SMTP server implementations.
4. After each login attempt my server responds with “Authentication unsuccessful.” After receiving this response, the attacking server resets the TCP connection and then establishes a new connection where a different username and password combination is used.

This pattern looped for 1.5 hours until I noticed it and blocked the IP. In that time, they had hit my server with almost 700 login attempts, each using a different username / password combination.

Typically mail servers will reject a RCPT TO address which would result in them being a relay, unless the sender is authenticated. If this attacker discovered a login that was accepted by my server, they very quickly would have turned my Exchange server into a relay to blast out spam and malicious messages.

Even with technologies like SPF, DMARC, and DKIM, spoofing and phishing attacks are still extremely prevalent. Some of these attacks can be stopped with properly configured SPF, DMARC, and DKIM, where as others need to be targeted with end user education.

This isn’t so much of a how to article, but more of a document to help identify various types of spoofing and phishing attacks, and what technologies you can use to stop them.

Header Terminology

The following excerpt from a message header will be used to identify the various headers that are important when dealing with a spoofing attack.

 Authentication-Results: spf=none (sender IP is 167.220.24.220)
 smtp.mailfrom=contoso.com; microsoft.com; dkim=none (message not signed)
 header.d=none;microsoft.com; dmarc=none action=none
 header.from=tailspintoys.com;
 From: From header FromHeader@tailspintoys.com    <-- 5322.From header
 To: Andrew Stobart andrew.stobart@microsoft.com
 Subject: Different MailFrom and From headers
 Return-Path: MailFromHeader@contoso.com    <-- 5321.MailFrom
 Reply-To: replyto@adventure-works.com   <-- Reply-To address

5321.MailFrom header

The MailFrom address represents the address that was sent in the envelope of the message. The domain used in this address is what the SPF check will be done against. In an SMTP conversation, this will be specified as MailFrom. In a message header, this will be identified as the Return-Path.

5322.From

This is the From address that you see in your mail client.

In the message header, this address will be the From header.

In most scenarios, the From and MailFrom headers will match, however these will typically be different in messages where spoofing is used (they also tend to be different for bulk mail). Typical spoofing attacks will only spoof the From header because SPF will not check the domain specified in this header, therefore bypassing SPF).

Authentication-Results header

This header will provide the results of the SPF/DKIM and DMARC checks. It also contains the following.

Smtp.mailfrom = Specifies the domain used in the 5321.MailFrom address. Header.from = Specifies the domain used in the 5322.From address. SPF = Results of the SPF check. DKIM = Results of the DKIM check. DMARC = Results of the DMARC check.

Reply-To header

If this header is set, when a mail client replies to the message, the reply will be directed to this address, regardless of what the original From or MailFrom addresses were.

Spoofing methods

The following are examples of the most common type of spoofing attacks.

5322.From header is spoofed, but the 5321.MailFrom is for a domain controlled by the attacker

Consider the following SMTP commands for a sent message:

 S: Helo woodgrovebank.com
 S: Mail from: phish@phishing.contoso.com  <-- 5321.MailFrom
 S: Rcpt to: astobes@tailspintoys.com
 S: data
 S: To: "Andrew Stobes" <astobes@tailspintoys.com>
 S: From: "Woodgrove Bank Security" security@woodgrovebank.com  <-- 5322.From

The recipient would see the message being sent from security@woodgrovebank.com.

The SPF check will be done against phishing.contoso.com, which will pass (or be none) because the attacker will own this domain. This is why this type of attack is so highly used. This type of spoof can be identified when you see a Return-Path header that is different from the From header. Here’s an excerpt of the message header for the above message.

 From: Woodgrove Bank Security <security@woodgrovebank.com>
 To: Andrew Stobart <andrew.stobart@microsoft.com>
 Subject: Different MailFrom and From headers
 Return-Path: phish@phishing.contoso.com    <-- SPF only checks this domain

In these types of messages, the attacker will usually not expect the user to reply back to them (as the message would go to the spoofed address which the attacker will not control). These types of messages typically contain a URL to a phishing site which the attacker is hoping the recipient will click on.

SPF combined with DMARC would detect this type of spoof. SPF on it’s own would not as SPF only checks the domain in the 5321.MailFrom address.

5322.From header spoofed, and 5321.MailFrom = 5322.From

Similar to the above attack, but in the headers you will see the From header and Return-Path header match. These attacks typically contain phishing links that the attacker hopes will be clicked on.

SPF will detect this attack; however, it is strongly recommended to implement DMARC along with SPF.

Homoglyph Attacks (attacker owns a domain that is similar to the customers)

These types of attacks are similar again to the above, except the domain being spoofed is owned by the attacker and is very similar to the company that they are attacking. Consider the following:

In this case, the attacker hopes that when the recipient replies back to the message, they will not notice that the domain they are replying to isn’t actually their own. We also see a similar type of attack used where the attacker spoofs the customers’ domain exactly, but then sets a Reply-To header which is set to a domain that is similar to the customers, but owned by the attacker.

Attacks that use the Reply-To header

When a Reply-To header is present, when a recipient replies back to a message, the reply will be directed to the address specified in the Reply-To header.

Example. Consider the following spoofed message. Notice how even the profile pictures appears.

Here’s what happens when I hit the Reply button.

Notice that the reply is being sent to a domain that the attacker will own. The attacker hopes that the recipient won’t notice that the reply to address is different from the sending address.

Here’s what the header looks like for this type of attack.

From: <name redacted>@microsoft.com
To: <name redacted>@microsoft.com
Reply-To: joe.someone@contoso.com
Subject: Wire transfer request
Return-Path: <name redacted>@microsoft.com

SPF and DMARC will detect these types of attacks. In addition, end user education is also highly recommended.

Anti-spoofing Technologies

Sender Policy Framework (SPF)

Sender Policy Framework or SPF is a technology for sender authentication. SPF uses a DNS TXT record to store a list of authorized sending IP addresses for a given domain. A receiving mail server performs an SPF check by confirming that the connecting IP address is in the SPF record for the 5321.MailFrom domain. In Exchange Online the results of the SPF check is recorded in the Authentication-Results header and Received-SPF header.

Because SPF only checks the domain in the 5321.MailFrom address, SPF cannot detect spoofing attacks where the 5322.From address is spoofed, and the 5321.MailFrom address is set to a domain that the attack owns, and can therefore ensure SPF passes. This is where DMARC comes in.

Domain-based Message Authentication, Reporting an Conformance (DMARC)

DMARC uses Sender Policy Framework (SPF) and/or DomainKeys Identified Mail (DKIM) to authenticate the 5322.From address. DMARC will pass if an SPF or DKIM check passes and the domain in the 5321.MailFrom and 5322.From addresses match. This means that DMARC will always fail if the domain of the 5321.MailFrom does not match the domain of the 5322.From.

DomainKeys Identified Mail

With SPF records, you can only add up to ten DNS lookups in the record. If you have many third parties that can send on your behalf, you will quickly hit this wall. This is where DKIM comes in. DKIM allows you to sign outbound mail with a signing certificate. The public key for the certificate is present in your Public DNS, which will allow recipients to verify the sender is authorized.

User education

End user education is an extremely important piece of preventing spoofing attacks. End users should be educated on the following.

• Hover over URLs in your email client to verify they are going to a known location
• Don’t click on URLs from people you aren’t familiar with
• When replying back to a message, always verify the expected address is present on the TO line

Resources

Specifications

• SPF
• DMARC
• DKIM

Microsoft blogs

• Using DMARC to prevent Spoofing
• Using DMARC in Office 365
• Best Practices for Exchange Online Protection customers to align with DMARC
• Enhanced email protection with DKIM and DMARC in Office 365
• Manually hooking up DKIM signing in Office 365
• Exchange Online is rolling out default DKIM-singing to everyone

YouTube video by Microsoft Support Escalation Engineers

It’s the last day of 2015 and I thought it would be interesting to look back on the top blog posts of this year. When thinking about this past year, the first topic that comes to mind is Spoofing/Phishing, and anti-spoofing technologies like SPF, DMARC, and DKIM. These types of attacks continue to be heavily used in the messaging world. From a feature perspective, we introduced Exchange Online Advanced Threat Protection, in addition to enabling support for DKIM signing on outbound mail. It’s been a busy year, and I’m sure 2016 will be even busier.

Here are the top posts from this blog for the last couple of years.

Top blog posts from 2015

1. Using DMARC to Prevent Spoofing
2. Parsing an Extended Message Trace
3. Useful Wireshark Filters for Mail Flow Troubleshooting
4. EOP Mysteries Solved – Mail Queuing in EOP Which is Destined On-premises
5. Scheduling Mail Reports in Office 365
6. Troubleshoot a Broken Junk Mail Folder
7. Ensure your SPF Record is Correct
8. Find the sending client IP for messages sent from an Exchange Online mailbox
9. An Introduction to the new Spam Filter Allow and Block Lists
10. Recipient Notifications

Top blog posts from 2014

1. Outbound Connector Smart Host Behavior
2. P2 Headers Now Respected for End User Safe and Blocked Senders Lists
3. TLS, Connectors, and You
4. Prevent DLs From Receiving ESNs
5. Quarantine and PowerShell
6. On-Premises Delivery Failover
7. Is X-Microsoft-Antispam a New EOP Header
8. Remote Desktop Connection Manager updated to v2.7
9. The Resolver
10. Verifying ESN Delivery

I hope you have had a great holiday with friends and family, and have a wonderful 2016!

The following article was written by Richard Deprez who is a Support Escalation Engineer for Exchange Online at Microsoft.

Recently I have seen an increase in phishing attacks coming through EOP due to safe sender aggregation (email addresses that have been added by end users to their safe senders list will be respected by EOP, that is, messages from these addresses will get an SCL of -1 when the message is destined to that particular user).  While everything is functioning as intended this can be a pain point if the email addresses that have been added are being spoofed by an attacker.

I also often see users adding internal email addresses to their safe senders list. In Office 365, authenticated mail will not be scanned by the spam filter, so there is no need to have any internal recipients on a user’s safe senders list.

In both of the above cases, an administrator will often want to remove specific addresses from an end user’s safe senders list. Editing the safe/block sender list for an individual is relatively simple: How to set up safe senders and blocked senders in Office 365.  However, attempting to find and remove in bulk can be a more difficult process due to the way the safe sender list is stored.

The below script (which is not officially supported by Microsoft) is a two-step process to identify and export the users who have a specific domain in the safe sender list and also to remove it from these users.  This way the users can be listed and educated about proper reasons for safe listing an address and removed to help prevent spoofed messages from coming in.

Step 1: Get the list of mailboxes who have the domain configured in their safe sender list:

get-mailboxjunkemailconfiguration * -resultsize unlimited|where {$_.TrustedSendersAndDomains -like ‘*@contoso.com’} |Export-csv ‘C:\testsafe.csv’ -notypeinformation

Step 2: The actual removal of the domains. In this scenario we are using the input file created in step 1; the script can be easily altered to just check all mailboxes and remove where applicable.

$newarray = @()
 $users = Import-csv ‘C:\testsafe.csv’
 foreach ($Identity in $users ){
 $temp = (Get-MailboxJunkEmailConfiguration -identity $Identity.Identity).TrustedSendersAndDomains
 foreach ($obj in $temp) { if (!(($obj -like '*@contoso.com'))) { $newarray += $obj }}
 Set-MailboxJunkEmailConfiguration -identity $Identity.Identity -TrustedSendersAndDomains $newarray
 $newarray = @()
 }

The above script has worked great for many of organizations I have worked with, but is not officially supported by Microsoft. Please ensure you test all your PowerShell scripts before running against production objects.

- Richard Deprez

It’s been just over two months since my last story and about time I post an update here. Since early January I’ve been on a medical leave from work which is the reason this blog has become so quiet. I’m not sure when I’ll be back at work yet, but when I do I’ll be posting here again and will catch up on everything that I’ve missed.

I hope Exchange Online has been treating you well while I’ve been gone. Rest assured, I will be back again soon!

Hi all, thank you for the kind comments on my last post where I talked about being away from work. I’m back at work now and will be posting to this blog again on a regular basis. I’ve missed writing here and am excited to resume the knowledge sharing and discussions.

There is a bit of a backlog of comments on this site. I’m going to try and respond to them, but it may be a bit with the current backlog that’s on my plate. Watch for a new post this week.

Cheers,

Andrew

The following article was written by Rob McCarthy who is a Business Program Manager for Readiness in Microsoft.

Spoofing continues to be a top concern for any security conscious email administrator today. Generally, there are two spoofing scenarios that administrators must devote thought to…

1. An organization is receiving spoofed messages from a malign source asking for network credentials. These emails must be blocked immediately. This is the one that keeps them up at night.
2. A legitimate newsletter sent from a 3rd party partner is being mistakenly marked as spam because it appears to be spoofing the organization. Users must receive these.

In an effort to provide transparency into spoof detection, and to more easily address both the false-negative and false-positive impact circumstance, a ‘Spoofed Mail Report’ will now be part of Exchange Online’s report catalogue.

The “Spoofed Mail Report” will not only provide administrators a clear view of specific spoofing instances and anomalies, it will allow the administrator to drill down into the detail of individual instances. Furthermore, administrators will be offered the ability to immediately block or allow these true senders based on their IP.

This new report can be found under the Protection Reports heading.

For additional information on this report, as well as the other reports, please see “Use mail protection reports in Office 365 to view data about malware, spam, and rule detections”

Your feedback shapes our products so please take a look and let us know what you think!

– Rob McCarthy

After being out of the office for four months, there is a lot I need to catch up on! Over the past couple of weeks, I’ve been reading through articles going over what all has changed in Exchange Online and Office 365 during my time away (January to May of this year).

I have complied a list of the most interesting articles that I’ve read from the past four months and wanted to share them here. Most of these are related to Exchange Online, but there are some PowerShell ones as well.

Blogs

• Office 365 is expanding its DKIM-signing to our consumer brands plus adding default signatures to enterprise email traffic (January 2016)
• Common errors in SPF records (February 2016)
• How antispoofing protection works in Office 365 (February 2016)
• A PowerShell script to help you validate your DKIM config in Office 365 (March 2016)
• New reporting portal in the Office 365 admin center (March 2016)
• A new tool for helping resolve Office 365 issues (May 2016)
• New enhancements to Office 365 eDiscovery (May 2016)

TechNet

• Use DKIM to validate outbound email sent from your domain in Office 365 (May 2016)
• How Office 365 uses Sender Policy Framework (SPF) to prevent spoofing) (May 2016)
• Setup SPF in Office 365 to help prevent spoofing (May 2016)
• Removing a user, domain, or IP address from a block list after sending spam email (March 2016)
• Use the delist portal to remove yourself from the Office 365 blocked senders list (April 2016)

PowerShell

• PowerShell Basics: (Part 1) Getting Started with PowerShell (February 2016)
• PowerShell Basics: (Part 2) Real World Examples for Our Most Common Cmdlets (February 2016)
• PowerShell Basics: (Part 3) Real World Script Examples (March 2016)
• PowerShell for Office 365 (This is from last year, but a very good resource)

Cheers!

Andrew

Environment

The Problem

Possible Solutions

1. Using the Exchange Server install media, extend the local Active Directory schema. I don’t recommend this and so will not go into details here. From a high level, this would add Exchange attributes to the local Active Directory. These attributes could then be set, and Azure AD Sync would then be configured to sync these attributes to Office 365.
   
   This option requires much testing, and there is always risk associated with AD schema changes.
   
   
2. The second option is to break the connection between the Office 365 user and their object in the Local Active Directory. One way of accomplishing this is to move the user object in the local Active Directory to an OU that does not synchronize to Office 365. Once the connection is broken, the cloud user can be modified in Office 365 as they are not tied to the local AD anymore.
   
   This method may or may not be feasible, depending on how permissions are setup and what Contoso’s security policies are. Something else to note is that breaking this connection will also break password synchronization from on-premises to Office 365.

In the end Contoso decided to go with neither of the above options. Instead they opted not to hide this user from the Exchange Online GAL.

Resources

I have recently found that many organizations are unaware that they can take action on their own when it comes to blocked IPs and banned senders in Office 365. For both of these cases, a web based portal exists that can be used to remedy the problem, eliminating the need to create a support case with Microsoft.

I have an IP that is currently blocked by Office 365

An organization that is trying to send mail to an Office 365 recipient will receive a response similar to the following if their sending IP is on the Office 365 block list.

550 5.7.606-649 Access denied, banned sending IP [IP address]; To request removal from this list please visit https://sender.office.com/ and follow the directions. For more information please go to http://go.microsoft.com/fwlink/?LinkID=526653.

To request delisting, anyone (even non-Office 365 customers) can visit https://sender.office.com/.

More information can be found on the Use the delist portal to remove yourself from the Office 365 blocked senders list TechNet page.

I have an end user that is on the Office 365 banned senders list

A user that sends a large amount of email from Office 365 that is classified as spam by the service will be blocked from sending messages. This typically happens when an Exchange Online account has become compromised. A user in this state will receive an NDR (Non-Delivery Report) when they attempt to send an email message.
An Office 365 tenant administrator can lift the outbound ban for their users from the Action Center which is located under Protection in the Exchange Online Admin Center.

Just search for the blocked account and click Unblock Account. Note that there is a limit to the number of times an unblock can be performed in the tenant in a specific time frame. If this limit is reached, then an error will be thrown and a support case will need to be opened.

More information on this can be found on the Removing a user, domain, or IP address from a block list after sending spam email TechNet page.

Wrapping things up

If you find yourself in one of the above two scenarios, I would highly recommend first visiting the appropriate web portal. This will often lead to the quickest resolution. For any problems with the procedures outlined on TechNet, or if you would just like to chat with a friendly individual, you can of course open a ticket with Microsoft support.

Resources

• Use the delist portal to remove yourself from the Office 365 blocked senders list
• Removing a user, domain, or IP address from a lock list after sending spam email

Creating custom management roles can be very powerful, and they not nearly as complicated as one might think. Most clicks in Office 365 actually cause a PowerShell cmdlet to run in the background. With a custom management role, you can specify right down to which cmdlets you would like a user to have access to. Technically, you can go right down to which parameters on a cmdlet that a user has access too, but I won’t be going that deep in this article.

Real world problem

I recently worked with an organization that wanted the following user role permissions for their end users.

• End users cannot create new distribution groups
• End users can add or remove users from distribution groups that they own

The above permission set is very reasonable; however, it is not possible to assign these permissions with the built-in management roles in Exchange Online.

Let’s look at the Default Role Assignment Policy which can be found in the Exchange Online portal under Permissions –> User Roles.

We have a management role that we can grant called MyDistributionGroups.

MyDistributionGroups grants permissions for distribution group creation, along with permissions allowing modifying distribution groups that users may own. With this default permission, it’s either all or nothing. Either end users can manage distribution groups they own and can create new distribution groups, or they cannot manage distribution groups that they own and they cannot create new distribution groups.

From this screen, we are unable to remove the ability of creating a new distribution group, while at the same time keeping the permission which allows distribution group owners to modify their groups.

Luckily, we have PowerShell and can create a new custom management role which will grant the permissions that we would like.

Background

Before we create the custom management role, let’s first talk about what we are going to accomplish from a high level.

For most mouse clicks that happen in Office 365, a PowerShell cmdlet is run in the background. Through the magic of a custom management role, we can limit which cmdlets can be run by users. If you want to get even more granular, you can restrict what parameters can be used with a cmdlet. Awesome right?

What we are going to do here is create a new management role which will initially be identical to the MyDistributionGroups role. We are then going to revoke access to the New-DistributionGroup cmdlet from this new management role. Once completed, we will then assign this new management role to our end users. This will allow end users to manage distribution groups that they own, but they will not be able to create new distribution groups because we have revoked their access to the New-DistriubtionGroup cmdlet.

Now let’s see how we accomplish this.

Steps

Let’s first see what cmdlets are granted when we assign the MyDistributionGroups management role by running the following.

(Get-ManagementRole MyDistributionGroups).roleentries

One of the cmdlets granted by the MyDistributionGroups role (and which we can see in the above list) is New-DistributionGroup. This is the cmdlet that we want to remove access to for end users.

Next we are going to create a new management role that we can edit. This new role will initially be a copy of the existing MyDistributionGroups role. The -Parent parameter in the command below is used to specify which existing management role we want to copy into our new management role. I’m calling the new management role No create DG.

New-ManagementRole -Name "No create DG" -Parent MyDistributionGroups

If we run (Get-ManagementRole “No create DG”).roleentries we will see that our new management role does indeed contain all of the role entries that the MyDistributionGroups management role contains.

Next, I’m going to search this new management role for the cmdlet that I want to remove. In this case the cmdlet we want to remove is New-DistributionGroup. I’m running this to verify that the New-DistributionGroup cmdlet is present in our new management role.

Get-ManagementRoleEntry "No create DG\*" | Where { $_.Name -Like "New-DistributionGroup" }

The output above confirms that New-DistributionGroup exists in our new management role.

Before we remove it, I like to run the remove command with the -WhatIf switch. This will allow me to verify that the command will indeed remove only the cmdlet that I want removed, while not actually taking any action.

Remove-ManagementRoleEntry "No create DG\New-DistributionGroup" -WhatIf

This output is exactly what I wanted to see. The New-DistributionGroup cmdlet will be removed from this group using this command. Next I re-run the same command, but this time without -WhatIf

Remove-ManagementRoleEntry "No create DG\New-DistributionGroup"

Click “yes” when prompted by PowerShell to confirm. We don’t see any errors here and so we assume the command successfully removed the New-DistributionGroup cmdlet from our new management role. Let’s double check though by searching our new management role, No create DG, for New-DistributionGroup.

We get no results, which confirms that the New-DistributionGroup cmdlet was successfully removed from our new management role, No create DG.

Applying our new Management Role

Before applying our new management role, let’s first look at what our end user sees in their Office 365 portal when they manage their distribution groups.

Here we can see this user is the owner of one distribution group. They also have a plus icon which will allow them to create a new distribution group.

Now let’s apply our new management role to this user. To do so, I’m going to change the permissions on the Default Role Assignment Policy user role. For testing, you instead may want to create a new user role with the new management role, and then assign this user role to a test user.

For my testing, I’m going to modify the Default Role Assignment Policy. In modifying this user role, notice how I now see my new management role listed under the MyDistributionGroups management role. I’m going to uncheck MyDistributionGroups, and then check off the management role I created and modified, No create DG.

This will impact my test user as they are currently assigned the Default Role Assignment Policy. After saving my update to the Default Role Assignment Policy and waiting about ten minutes, I refresh the view of my end user and this is what they now see.

The plus sign in their portal view has now been removed. It’s like magic! They can no longer create new distribution groups. However, they can still modify distribution groups that they are the owner of by clicking the pencil icon which is still present.

The organization I worked with went a step further and also removed the Remove-DistributionGroup cmdlet from their custom management role. This caused the trash can icon seen above to also disappear from the end users view, meaning they could also not delete distribution groups.

Summary

Custom management roles in Exchange and Exchange Online are very powerful and allow for the creation of very granular permissions. Fire up that PowerShell and try them out!

Resources

• New-ManagementRole cmdlet
• Remove-ManagementRoleEntry cmdlet
• Remove Multiple Management Role Entries in Office 365

The Common Attachment Types Filter is a feature that was rolled out to Exchange Online earlier this year. If you haven’t opened your malware filter for a while, you may not even know this new filter is there!

With this filter, you no longer need to create transport rules to block file types as attachments (for example, .exe, .reg, .bat, etc…). While attachments are always scanned for malware, organizations typically prevent certain file types from entering their environment through email.

If your organization currently uses transport rules to block certain file types, moving that configuration over to the Common Attachment Types Filter can provide benefits like user notifications and reducing your number of transport rules.

If you were an Exchange Online customer prior to the roll out of this filter, by default it will be disabled. For organizations that signed up for Exchange Online after this filter was rolled out, you will find this filter enabled by default.

Enabling and Configuring

The Common Attachment Types Filter can be found in the malware filter. In the Exchange Online portal, browse to Protection, and then Malware Filter.

In the settings of the malware filter, you will find the Common Attachment Types Filter configuration.

By default the File Types list will be pre-populated with file extensions that can be dangerous. This list can of course be modified to your hearts content.

With transport rules, it’s not possible to strip an attachment while still allowing the message to be delivered. Because the Common Attachment Types filter is part of the malware filter, this is possible by enabling an end user response.

With the above response enabled, if a user receives a message with a file attachment type that is on the filter list, the attachment will be replaced with a text file which contains information about the block. The body of the original message will remain intact and be delivered along with the replacement text file. This is an example of a text file that would replace a stripped attachment.

Another benefit is that this is a True-Type filetype filter. This means that we try to detect the file type based on its contents, not just by the extension. For example, if an executable file has its extension renamed from .exe to .txt, it will still be detected as an executable file and stopped.

Summary

If you have been using transport rules to prevent certain attachment types from being sent, I would recommend you check out the Common Attachment Types Filter in the malware filter. There are many benefits from using this filter to block file types as opposed to transport rules.

Please let me know what your thoughts are in the comments below.

Resources

• Configure anti-malware policies
• Leading the way in the fight against dangerous email threats

Rather than tease you with a witty, or even humorous opening paragraph, I’m going to instead jump right to the dessert. Because really, who doesn’t love dessert?

As I recently discovered with an organization, adding disclaimers to calendar invite emails can cause problems and confusion for end users. When an individual shares their calendar, we would expect the recipient to the see follow invitation in Outlook.

The recipient simply clicks “Open this Calendar” and magic happens.

In OWA, the received sharing invite would look like this.

The user simply clicks “Accept.”

The problem arises if a transport rule tries to add a disclaimer to this inbound sharing invitation. First, some background. If a transport rule cannot add a disclaimer, the default action it takes is to wrap the message.

This means that the original message will be attached as a .eml file to a new message which will contain the disclaimer.

Here’s what the same calendar invite looks like in the desktop Outlook client after a transport rule has wrapped the message to add a disclaimer.

There’s no “Open this Calendar” button here because the original invite is the attachment. The recipient will need to first open the attached message, and from there they can open the shared calendar. Although this is more clicks, the recipient can still open the shared calendar.

The OWA experience is not so… user friendly. In OWA, the recipient will see the attachment, but unfortunately OWA can’t open .eml files. This can be seen below.

The user could download the .eml file and try to open it from their computer, but they have most likely already called their help desk.

Resolution

The best course of action here is to not add disclaimers to calendar invitation emails. This can easily be done by adding an exception clause to the existing transport rule. The transport rule exception would contain, “Except if… The subject or body includes…”

• has invited you to view his or her Microsoft Exchange Calendar
• Sharing invitation
• Sharing request
• I’d like to share my calendar with you

Here’s how our modified transport rule would look after the above exception was added.

With this in place, the transport rule will no longer trigger on calendar invitation emails, and thus leaving the end user with a positive experience (we hope!).

I recently found a new option you can select when releasing a message from the quarantine. While this option isn’t brand new, I’m not sure when it was added, and so wanted to share for visibility. The TechNet article that references this option is dated from June of this year, so I’m guessing it was added around then.

When an administrator is releasing a message from the quarantine (through the Exchange Online portal), there is a “new” link they can click.

Release selected message and allow sender.

It appears both on this screen:

And on this screen:

If this link is clicked, the message will be released (obviously), and the sender will be added to the Sender Allow List, which is located in the Spam Filter in the Exchange Online portal.

In my testing, if you have multiple spam filters, the sender will be added to the Sender Allow List in each of them.

An end user that will not see this option if they log into their online quarantine view. The option to release and safe list is only available in the Exchange Online portal.

One thing to note here, if the message was quarantined because of a transport rule or blocked sender, the sender will continue to be blocked for future messages.

Resources

• Find and release quarantine messages as an administrator
• Find and release quarantine messages as an end user

With my move to special projects this year, I wasn’t able to post as many articles as I would have liked to. With that being said, I’m starting to do more case work again and will be trying to publish more articles in 2017. If you have any ideas for content or have specific questions, add them to the comments here and I’ll either respond in the comments or write an article about the topic.

I hope you all have a wonderful holiday season, and I’ll see you again in the new year!

Cheers,

Andrew

It’s a new year, and that means it’s time to look back at the top posts on this blog for last year. 2016 was a slower year for EOP Field Notes as I was on a leave from work for the first 5 months of the year. I returned in May when I continued my writing here.

Looking at the top posts, those with content on spoofing received the most views. Over the past year, there has been much more awareness about spoofing, and preventative technologies like SPF, DMARC, and DKIM.

You’ll notice that my top post from 2016 was actually published on December 23, 2015, but I’m including it in this list because it received most of its views in 2016. I’ve also included previous years top ten list.

Top ten posts of 2016

1. Troubleshooting and Identifying Spoofing Attacks
2. Bulk editing of Safe Sender lists
3. Take Action Yourself on Blocked IPs and Banned Senders in Office 365
4. Introducing Spoof Mail Reports
5. The Common Attachment Types Filter
6. Create a Custom Management Role for Granular Permissions in Exchange Online
7. Interesting case where Exchange is not installed on-premises
8. Interesting Exchange Online articles from the year so far
9. Disclaimers and calendar invites
10. Release from quarantine and safe list the sender in one click

Top ten posts of 2015

1. Using DMARC to Prevent Spoofing
2. Parsing an Extended Message Trace
3. Useful Wireshark Filters for Mail Flow Troubleshooting
4. EOP Mysteries Solved – Mail Queuing in EOP Which is Destined On-premises
5. Scheduling Mail Reports in Office 365
6. Troubleshoot a Broken Junk Mail Folder
7. Ensure your SPF Record is Correct
8. Find the sending client IP for messages sent from an Exchange Online mailbox
9. An Introduction to the new Spam Filter Allow and Block Lists
10. Recipient Notifications

Top ten posts of 2014

1. Outbound Connector Smart Host Behavior
2. P2 Headers Now Respected for End User Safe and Blocked Senders Lists
3. TLS, Connectors, and You
4. Prevent DLs From Receiving ESNs
5. Quarantine and PowerShell
6. On-Premises Delivery Failover
7. Is X-Microsoft-Antispam a New EOP Header
8. Remote Desktop Connection Manager updated to v2.7
9. The Resolver
10. Verifying ESN Delivery

Is there anything in particular, that you would like to see an article written about? Let me know in the comments.

Cheers,

Andrew