https://old.reddit.com/r/sysadmin/comments/rt91z6/exchange_2019_antimalware_bad_update/ jump to content my subreddits edit subscriptions * popular * -all * -random * -users | * AskReddit * -funny * -gaming * -pics * -news * -movies * -todayilearned * -tifu * -explainlikeimfive * -mildlyinteresting * -nottheonion * -aww * -videos * -worldnews * -LifeProTips * -Jokes * -dataisbeautiful * -OldSchoolCool * -books * -Showerthoughts * -askscience * -IAmA * -science * -TwoXChromosomes * -Art * -sports * -gifs * -Music * -space * -gadgets * -food * -Documentaries * -Futurology * -nosleep * -DIY * -WritingPrompts * -GetMotivated * -UpliftingNews * -EarthPorn * -history * -InternetIsBeautiful * -photoshopbattles * -creepy * -philosophy * -announcements * -listentothis * -blog more >> sysadmin sysadmin * comments * other discussions (2) Want to join? Log in or sign up in seconds.| * English [ ][] [ ]limit my search to r/sysadmin use the following search parameters to narrow your results: subreddit:subreddit find submissions in "subreddit" author:username find submissions by "username" site:example.com find submissions from "example.com" url:text search for "text" in url selftext:text search for "text" in self post contents self:yes (or self:no) include (or exclude) self posts nsfw:yes (or nsfw:no) include (or exclude) results marked as NSFW e.g. subreddit:aww site:imgur.com dog see the search faq for details. advanced search: by author, subreddit... this post was submitted on 01 Jan 2022 1,162 points (98% upvoted) shortlink: [https://redd.it/rt91] [ ][ ] [ ]remember mereset password login ATF Submit a new text post Get an ad-free experience with special benefits, and directly support Reddit. get reddit premium sysadmin joinleave661,708 readers 2,172 users here now A reddit dedicated to the profession of Computer System Administration --------------------------------------------------------------------- Rules 1. Community members shall conduct themselves with professionalism. 2. Do not expressly advertise your product. More details on the rules may be found in the wiki. --------------------------------------------------------------------- For IT career related questions, please visit /r/ITCareerQuestions --------------------------------------------------------------------- Please check out our Frequently Asked Questions, which includes lists of subreddits, webpages, books, and other articles of interest that every sysadmin should read! Checkout the Wiki Users are encouraged to contribute to and grow our Wiki. So you want to be a sysadmin? RTFM --------------------------------------------------------------------- Sysadmin Jobs Official IRC Channel - #reddit-sysadmin on irc.libera.chat Official Discord - https://discord.gg/sysadmin --------------------------------------------------------------------- * Link Flair Filters * Gilded Comments a community for 13 years BTF MODERATORS * message the mods * Moderator list hidden. Learn More discussions in r/sysadmin <> X 327 * 166 comments Seriously....what is the RIGHT way to set up a print server these days? 186 * 34 comments University loses 77TB of research data due to backup error 1166 * 372 comments Exchange 2019 Anti-Malware - Bad Update? 63 * 16 comments Dear @msexchangeteam. The FIP-FS "Microsoft" Scan Engine Failed to Load. Can't Convert "2201010001" to long. 8 * 17 comments SSH client for iPad with multi tab 936 * 93 comments Last day is today. Thank you. 40 * 8 comments Finally back to sysadmin! 5 * 3 comments Prevent upgrade to Windows 11 while keeping automatic feature updates 5430 * 625 comments [short rant] My entire company has this entire week off, including IT. The sheer amount of people thinking that because they choose to work on their vacation means that I also need to be available to support them is ridiculous. 2 * 6 comments Nimble HF40 iSCSI issue in configuration checks Welcome to Reddit, the front page of the internet. Become a Redditor and join one of thousands of communities. x 1161 1162 1163 Exchange 2019 Anti-Malware - Bad Update? (self.sysadmin) submitted 20 hours ago * by FST-LANE[ey2iodron2][wat5e8yfjq]2 [platinum_4][80j20o397j][tcofsbf92m][5izbv4fn0m]17[Illuminati] [45aeu8mzvs][Animated_H]2[2jd92wtn25][silver_48]9[gold_48]6 [klvxk1wggf]11& 49 more EDIT: I can't change the title, but this appears to be more serious than a bad update. Read on.... https://www.neowin.net/news/ y2k22-bug-microsoft-rings-in-the-new-year-by-breaking-exchange-servers-all-around-the-world / ------------------------------------ Just wondering if any other Exchange admins had their new year's celebration interrupted due to the "Microsoft Filtering Management Service" being stopped and reports of issues with mail flow? In the application event logs, I see a bunch of errors from FIPFS service which say: Cannot convert "220101001" to long If I look back further in the logs, it appears like it all started happening when the "MS Filtering Engine Update" process received the "220101001" update version just over an hour ago at 7:57pm EST. EDIT: I've tried forcing it to check for another update, but it returned "MS Filtering Engine Update process has not detected any new scan engine updates". ... I've temporarily disabled anti-malware scanning, to restore mail flow for now. TL DR; Microsoft released a bad update for Exchange 2013 (?), 2016 and 2019. Disabling OR bypassing anti-malware filtering will restore mail flow in the interim. UPDATE: according to @ceno666 the issue also seems to occur with the 220101002 update version as well. Could be related to, what I'm dubbing, the "Y2K22" bug. Refer to the comment from JulianSiebert about the "signed long" here: https://techcommunity.microsoft.com/t5/ exchange-team-blog/ december-2021-exchange-server-cumulative-updates-postponed/bc-p/ 3049189/highlight/true#M31885 The "long" type allows for values up to 2,147,483,647. It appears that Microsoft uses the first two numbers of the update version to denote the year of the update. So when the year was 2021, the first two numbers was "21", and everything was fine. Now that it's 2022 (GMT), the update version, converted to a "long" would be 2,201,01,001 - - which is above the maximum value of the "long" data type. @Microsoft: If you change it to an 'unsigned long', then the max value is 4,294,967,295 and we'll be able to sleep easy until the year 2043! UPDATE: Microsoft has confirmed disabling the malware filtering is the correct course of action for now (workaround to restore mail flow). While new signatures and engine updates have been released, they don't seem to fix the issue. We'll continue to wait for an official response from Microsoft. At least we have a third-party filtering/scanning solution in front of Exchange. * 372 comments * share * save * hide * report top 200 commentsshow all 372 sorted by: best topnewcontroversialoldrandomq&alive (beta) [ ] Want to add to the discussion? Post a comment! Create an account [-]runningntwrkgeek 166 points167 points168 points 19 hours ago (5 children) Thanks /r/sysadmin! Because of this, I checked my onprem 2019 and discovered we are impacted by this. I'm now working on it before I get phone calls. * permalink * embed * save * report * give award * reply [-]EPHEBOX 77 points78 points79 points 8 hours ago (4 children) Once you've finished testing go ahead and send an "outage update" email to everyone. Potentially nobody noticed but it's free PR for the IT department. * permalink * embed * save * parent * report * give award * reply [-]runningntwrkgeek 41 points42 points43 points 8 hours ago (3 children) I sent one out at 12:30 am to the entire company. Started with a request that coworkers be extra cautious due to one layer of security being disabled, followed with a tldr, then a more detailed explanation. Sometimes it's nice to let management know that just because things are working, it doesn't mean I'm not doing anything * permalink * embed * save * parent * report * give award * reply [-]rebris 31 points32 points33 points 8 hours ago (2 children) "Hey my emails stopped coming in for a while there, did that have anything to do with the thing you sent? I think it might've fixed itself though. Also my phone isn't working" * permalink * embed * save * parent * report * give award * reply [-]weaponizedlinux 13 points14 points15 points 6 hours ago (0 children) Oh my god are you one of my users? * permalink * embed * save * parent * report * give award * reply [-]uuufffff 1 point2 points3 points 5 hours ago (0 children) I laughed a lot and cried a bit too, not gonna lie. * permalink * embed * save * parent * report * give award * reply [-]brianmarcotte 99 points100 points101 points 15 hours ago[gold_48]2 [klvxk1wggf]2 (10 children) Here's what I did: Check queues Get-Queue Set filter to bypass Get-ExchangeServer | % {Set-MalwareFilteringServer -BypassFiltering $true -Identity $_.Name} Restart Transport service $ExchangeServers = Get-ExchangeServer | Select -ExpandProperty Name $ExchangeServers | % {Get-Service -ComputerName $_ -ServiceName MSExchangeTransport | Restart-Service -Force} If ForEach takes too long, then just manually restart the service in services.msc Check queues Get-Queue Queues should start to process messages I'll check the malware filter upon my return to see if MS has addressed the issue with an update Get-ExchangeServer | Get-Queue To check queues on all transport servers. Just quick and dirty, but does the job for an urgent issue on New Year's Eve. Our servers are still behind a third-party mail filter, so I'm ok leaving this MS filter off until they get their shit together. * permalink * embed * save * report * give award * reply [-]killerpm/dev/null - No Escape 10 points11 points12 points 6 hours ago (1 child) Thanks! This saved me on a hungover new years day. Sigh.. haha * permalink * embed * save * parent * report * give award * reply [-]DeptOfOne 2 points3 points4 points 4 hours ago (0 children) Thank you my friend. I have been working on this for the past 3 hours. Did I mention I'm on Vacation and got called in for this? I owe you a drink my friend. * permalink * embed * save * parent * report * give award * reply [-]pauljdavis 2 points3 points4 points 5 hours ago (0 children) Great notes. That's an awesome comment - thanks for helping the community that way! Happy New Year! * permalink * embed * save * parent * report * give award * reply [-]guynamedjosh92 2 points3 points4 points 4 hours ago (1 child) Thank you! We tried this, but for some reason we have over 216k messages stuck in the queues of our 6 servers in the "Ready" status... Waiting on a Microsoft engineer to get on the phone with us to see why our queues aren't clearing fast enough (if it weren't for us receiving emails slowly, we'd say it's not fixed at all). * permalink * embed * save * parent * report * give award * reply [-]tourneytalk24 1 point2 points3 points 2 hours ago (0 children) See if you have any mail flow rules that check attachments. If you do, disable them and see if that clears it. * permalink * embed * save * parent * report * give award * reply load more comments (4 replies) [-]ceno666 50 points51 points52 points 19 hours ago (12 children) Just got the same problem here, are we sure it is a bad update or do we have a Year2k22 Bug situation? The FIP-FS "Microsoft" Scan Engine failed to load. PID: 7948, Error Code: 0x80004005. Error Description: Can't convert "2201010002" to long. * permalink * embed * save * report * give award * reply [-]ceno666 27 points28 points29 points 19 hours ago (6 children) check last comment, the plot thickens https://techcommunity.microsoft.com/t5/exchange-team-blog/ december-2021-exchange-server-cumulative-updates-postponed/bc-p/ 3049189/highlight/true#M31885 * permalink * embed * save * parent * report * give award * reply [-]FST-LANE[S] 67 points68 points69 points 19 hours ago* (5 children) TL DR; "a signed long is not big enough for a 6 digit date + 4 digits time or serial number". So for those of us who are too young to remember the Y2K scare, welcome to "Y2K22"! The "long" data type allows for values up to 2,147,483,647. It appears that Microsoft uses the first two numbers of the update version to denote the year of the update. So when the year was 2021, the first two numbers were "21", and everything was fine. Now that it's 2022 (GMT), the update version, converted to a "long" would be 2,201,01,001 - - which is above the maximum value of the "long" data type. @Microsoft: If you change it to an 'unsigned long', then the max value is 4,294,967,295 and we'll be able to sleep easy until the year 2043! * permalink * embed * save * parent * report * give award * reply [-]ceno666 15 points16 points17 points 19 hours ago (0 children) lol yeah i can remember, seems like it took 22 more years to get me called out of sleep for this shit * permalink * embed * save * parent * report * give award * reply [-]iamsplendid 25 points26 points27 points 17 hours ago (1 child) to detonate the year of the update The typo is so accurate for this * permalink * embed * save * parent * report * give award * reply [-]FST-LANE[S] 9 points10 points11 points 17 hours ago (0 children) Lmfao. Fixed. I think it was all those fireworks my neighbors were setting off! * permalink * embed * save * parent * report * give award * reply [-]disclosure5 13 points14 points15 points 18 hours ago (0 children) !remindme 21 years * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]FST-LANE[S] 11 points12 points13 points 19 hours ago (0 children) Well, you just confirmed that the newer 220202002 update is also broken. Maybe you're right; it's a Y2k22 bug. * permalink * embed * save * parent * report * give award * reply [-]JustSayTomato 9 points10 points11 points 17 hours ago (3 children) This is one of the first things I noticed when I started combing through Event Viewer. How in the hell did this ever get released? Especially on New Years Eve!? * permalink * embed * save * parent * report * give award * reply [-]FST-LANE[S] 7 points8 points9 points 17 hours ago (1 child) Probably automated. The version is based on the date in GMT. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) load more comments (1 reply) [-]brokenvcenter 245 points246 points247 points 20 hours ago [klvxk1wggf] (58 children) Hello friend. Same thing. Set-MalwareFilteringServer -BypassFiltering $True -identity Queues starting clearing. * permalink * embed * save * report * give award * reply [-]FST-LANE[S] 77 points78 points79 points 20 hours ago* (24 children) Thanks. Glad I'm not the only one. I did something similar to restore mail flow for now by running the Disable-AntiMalwareScanning.ps1 script from the Scripts folder in the Exchange install directory and restarting the transport service. * permalink * embed * save * parent * report * give award * reply [-]brokenvcenter 24 points25 points26 points 20 hours ago (8 children) Can confirm Disable-AntimalwareScanning.ps1 -forcerestart does the trick. 2013 seems to be unaffected. * permalink * embed * save * parent * report * give award * reply [-]FST-LANE[S] 34 points35 points36 points 20 hours ago (1 child) "Security by Antiquity" * permalink * embed * save * parent * report * give award * reply [-]brokenvcenter 15 points16 points17 points 20 hours ago (0 children) Availability by Antiquity! * permalink * embed * save * parent * report * give award * reply [-]TraditionalWealth293 6 points7 points8 points 13 hours ago (0 children) Can confirm this worked on Exchange 2016 CU22. Had to run it on all DAG members, FYI. * permalink * embed * save * parent * report * give award * reply [-]tranceandsoul 3 points4 points5 points 9 hours ago (3 children) Thanks!!! What are the risks of having this disabled? * permalink * embed * save * parent * report * give award * reply [-]the_bushman924 1 point2 points3 points 6 hours ago (0 children) https://docs.microsoft.com/en-us/exchange/ disable-or-bypass-anti-malware-scanning-exchange-2013-help The question we all want to know! I haven't disabled anything until this can be answered. Thankfully, I have understandable clients. * permalink * embed * save * parent * report * give award * reply [-]Snowman25_ 1 point2 points3 points 3 hours ago (1 child) Isn't the name pretty self-explanatory? * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) load more comments (1 reply) [-]UDP161 67 points68 points69 points 19 hours ago (5 children) THANK YOU. What in the absolute hell Microsoft!? On New Years Eve!? First place I check is Reddit and you guys save my life before we even get an engineer on the phone. Thank god for that premium support... * permalink * embed * save * parent * report * give award * reply [-]sykophreak 20 points21 points22 points 18 hours ago (2 children) I wish I'd checked Reddit first. I spent a good hour troubleshooting it and figured out the fix before checking here. * permalink * embed * save * parent * report * give award * reply [-]wgbeatty 1 point2 points3 points 7 hours ago (0 children) Same * permalink * embed * save * parent * report * give award * reply [-]KyAaron 1 point2 points3 points 2 hours ago (0 children) Spent 3 hours going insane before seeing this and fixing it in 5 minutes. Reminder to always check Reddit first. * permalink * embed * save * parent * report * give award * reply [-]BrFrancis 10 points11 points12 points 17 hours ago (0 children) In other news, FireEye ETP and EX don't do this... Nor does.. -check list- any other email anti-malware vendor... So... Happy new years * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]patrynmaster 12 points13 points14 points 17 hours ago (0 children) HANK YOU. What in the absolute hell Microsoft!? On New Years Eve!? First place I check is Reddit and & $env:ExchangeInstallPath\Scripts\Disable-Antimalwarescanning.ps1 * permalink * embed * save * parent * report * give award * reply [-]fluxboxuk 2 points3 points4 points 9 hours ago (0 children) Confirmed as working on Exchange 2016... MS premier support have confirmed its a known issue, but no known fix as yet ! * permalink * embed * save * parent * report * give award * reply [-]siedenburg2Sysadmin 2 points3 points4 points 11 hours ago (0 children) The one day I tought I could do nothing I had a feeling and visited this subreddit, luckily I've done that, else it would be really stressfull on the first workday of the year. * permalink * embed * save * parent * report * give award * reply [-]rhutanium 2 points3 points4 points 19 hours ago (0 children) Thanks, this fixed it for me. Happy New Year! * permalink * embed * save * parent * report * give award * reply [-]brokenvcenter 1 point2 points3 points 20 hours ago (0 children) Happy new year!! * permalink * embed * save * parent * report * give award * reply load more comments (4 replies) [-]pssssn 31 points32 points33 points 19 hours ago (1 child) Anyone coming across this, restart the Microsoft Exchange Transport service after setting this value. * permalink * embed * save * parent * report * give award * reply [-]Intros9Jack of All Trades 4 points5 points6 points 16 hours ago (0 children) Yep, this and the above command got us working again. Merry New Year! * permalink * embed * save * parent * report * give award * reply [-]dickielaw88 15 points16 points17 points 19 hours ago (3 children) I did this command, but my queue seems to be stuck. Any ideas how to get it moving again? Edit: After a restart the queue cleared. * permalink * embed * save * parent * report * give award * reply [-]its_the_revolutionIT Manager 10 points11 points12 points 19 hours ago (1 child) It takes up to 10 min to process https://docs.microsoft.com/en-us/exchange/ disable-or-bypass-anti-malware-scanning-exchange-2013-help "Bypassing or restoring malware filtering doesn't require you to restart any services. However, changes to the setting may take up to 10 minutes to take effect." * permalink * embed * save * parent * report * give award * reply [-]ComGuards 5 points6 points7 points 16 hours ago (0 children) But then running the script results in the following output (At least on 2016): WARNING: The following service restart is required for the change (s) to take effect : MSExchangeTransport Anti-malware scanning is successfully disabled. Please restart MSExchangeTransport for the changes to take effect. Classic Microsoft =P. * permalink * embed * save * parent * report * give award * reply [-]torbar203whatever 2 points3 points4 points 3 hours ago (0 children) https://www.reddit.com/r/sysadmin/comments/rt91z6/ exchange_2019_antimalware_bad_update/hqtt5ib/ I found a transport rule was keeping things stuck in the submission queue * permalink * embed * save * parent * report * give award * reply [-]Remarkable_Point_179 3 points4 points5 points 11 hours ago (1 child) Seems to apply to all versions of Exchange any CU, we have the latest patched, same issue, does look like a Y22 issues, disabling malware clears the queue and mail flows after transport restart, I am now working through all the exchange severs we support which is a lot. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]Pretend_Sock7432 1 point2 points3 points 14 hours ago (0 children) Disable-AntimalwareScanning.ps1 -forcerestart Thanks for this, had some very nice morning today. Just to add, restart also the transport service. * permalink * embed * save * parent * report * give award * reply [-]wewpo 0 points1 point2 points 17 hours ago (0 children) Thanks a heap for this. * permalink * embed * save * parent * report * give award * reply [-]Tkaranik 1 point2 points3 points 13 hours ago (0 children) Confirmed it fixed my Exchange 2016 queue. Many thanks * permalink * embed * save * parent * report * give award * reply load more comments (22 replies) [-]hack819 25 points26 points27 points 19 hours ago (1 child) I should have checked here before spending the last hour screwing with exchange. Thanks for the heads up. * permalink * embed * save * report * give award * reply load more comments (1 reply) [-]JeepMunkee 68 points69 points70 points 20 hours ago (3 children) Yep. All email stopped processing on my on prem 2016 at 645pm. I spent like 40 mins rebooting and stressing out. Thanks for your post! FML. lol * permalink * embed * save * report * give award * reply [-]FST-LANE[S] 38 points39 points40 points 20 hours ago (1 child) Leave it to big corporations like Microsoft to screw us at the most inconvenient time! Happy New Year! * permalink * embed * save * parent * report * give award * reply [-]JeepMunkee 6 points7 points8 points 20 hours ago (0 children) Happy Fucking New Year! * permalink * embed * save * parent * report * give award * reply [-]ThreshOP 1 point2 points3 points 16 hours ago (0 children) I'm gonna take a wild guess, you're GMT-6 timezone? * permalink * embed * save * parent * report * give award * reply [-]FrankyHugo 15 points16 points17 points 17 hours ago (1 child) Thanks MS for the Y2K22 bug, great job sleep is so overrated.Thanks guys for giving a sollution for the problem * permalink * embed * save * report * give award * reply load more comments (1 reply) [-]T101M850Director of Technical Services 13 points14 points15 points 19 hours ago (0 children) Popped some champagne and migrated from the desk to the couch to watch Miley and Pete with the wife...Cue slack exploding my phone with 800 messages. Finservice company, so year end is kind of a big deal. * permalink * embed * save * report * give award * reply [-]dickielaw88 44 points45 points46 points 20 hours ago (0 children) Our 2016 server is affected also. Damn Microsoft. Having this happen on new years eve? First Betty White, and now this. Hoping for a better 2022! * permalink * embed * save * report * give award * reply [-]pingsandchickenwingsSysadmin 12 points13 points14 points 18 hours ago (0 children) Amazing you saved my new years, friend. My appreciation. * permalink * embed * save * report * give award * reply [-]bugalouInfrastructure Architect 12 points13 points14 points 17 hours ago (7 children) This crap just brought down all of our alerting in the middle of new years eve. I am in the hospitality/entertainment industry so pretty much the worse time ever. * permalink * embed * save * report * give award * reply [-]FST-LANE[S] 16 points17 points18 points 17 hours ago (4 children) I was also a bit confused that my monitoring system was calling my phone when I hadn't seen any email alerts (it escalates to phone call if I don't acknowledge the alerts that comes through via email). But that's the downside of email alerts; if ANYTHING that the mail server relies on goes down, it takes email alerts down with it. My monitoring system calls a simple PowerShell script that I wrote which interacts with the Twilio API to call my cell phone and do some text-to-speech with the alert. On my cell phone, I set that contact to bypass "do not disturb" mode and a custom alarm ringtone, so even when I'm hibernating, it will wake me / give me a heart attack. * permalink * embed * save * parent * report * give award * reply [-]anonymous_commentor 2 points3 points4 points 3 hours ago (2 children) Check out Mailive. External, you set up a forwarding rule and it watches round trip times. Completely external. * permalink * embed * save * parent * report * give award * reply load more comments (2 replies) load more comments (1 reply) [-]Bleakbrux 2 points3 points4 points 3 hours ago* (0 children) Yeah my alerting too. We are office365 Exchange online apart from on premises alerts. I didn't notice until like 4pm that there had been zero alerts from anything. Only noticed as Veeam backup notifications didn't hit the mailbox and Firewall port scan Alerts were non existent which never happens. Thought it was Just blissfully quiet. Turns out There was a Microsoft induced shit storm going down. Should of known better. Thank god for exchange online and mimecast. It's nice to know these days that an exchange VM going pop only really affects IT. I didn't get the alert to say the alerting had gone down, clearly. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]HJForsythe 34 points35 points36 points 16 hours ago* (9 children) It's good that they just raised licensing pricing 10% again. Maybe they can hire some people that aren't fucking idiots with the extra money. I don't see a single mention of this anywhere official on Microsoft's end. Did they seriously push out this update and then leave for the weekend? What is happening at that company? I can't really put into words how hopeless it feels to be someone responsible for running this stuff. * permalink * embed * save * report * give award * reply [-]disclosure5 11 points12 points13 points 13 hours ago (4 children) I don't see a single mention of this anywhere official on Microsoft's end. I definitely remember these sort of things when people tell me they want to use products like this so they can get stellar MS support. I can near guarantee if I logged paid support case this morning they would have emailed and asked for logs and then be weeks away from responding again. * permalink * embed * save * parent * report * give award * reply [-]xirsteon 15 points16 points17 points 11 hours ago (1 child) Kindly do the needful, run the Healthchecker.ps1 script, send us the report and REVERT back. * permalink * embed * save * parent * report * give award * reply [-]bill-m 2 points3 points4 points 3 hours ago (0 children) This comment needs a trigger warning. Bad flashback. * permalink * embed * save * parent * report * give award * reply [-]172pilotsteve 6 points7 points8 points 8 hours ago (1 child) We have premier support and opened a SEV-A at about 04:00 GMT.. They did confirm that it was a systemic problem. We were able to figure out the workaround on our own, but after 2 hours they did provide the same recommendation. I still have the ticket open with them but haven't received another update from them.. I'm expecting a call in the next hour if they follow through with what they told me. * permalink * embed * save * parent * report * give award * reply [-]SheeEttinSysadmin 3 points4 points5 points 4 hours ago (0 children) They did confirm that it was a systemic problem. What is, their product quality issues? * permalink * embed * save * parent * report * give award * reply [-]TumsFestivalEveryDay 1 point2 points3 points 8 hours ago (0 children) Especially considering there's been some really unacceptable EXOL downtime lately and they never determined any root cause other than their "we dunno, we're examining our useless telemetry to make sure this doesn't happen again" hand-wave. * permalink * embed * save * parent * report * give award * reply load more comments (3 replies) [-]MarkDePalma 10 points11 points12 points 20 hours ago (8 children) Got an email flow alert, spent 5 min looking into it and saw the same. Disabling the engine like the others stated (either option) works. * permalink * embed * save * report * give award * reply [-]pssssn 4 points5 points6 points 19 hours ago (6 children) What are you using for this btw? I have a solution, but they don't have the option to repeat alerts, which is annoying. * permalink * embed * save * parent * report * give award * reply [-]MarkDePalma 7 points8 points9 points 18 hours ago (4 children) MX Toolbox integrates into our primary monitoring solution (LogicMonitor). We do external round trip time testing there and then have additional monitors on the transport queues and other things. * permalink * embed * save * parent * report * give award * reply load more comments (4 replies) load more comments (1 reply) load more comments (1 reply) [-]the_bushman924 9 points10 points11 points 6 hours ago (7 children) Serious question though. Why are you guys feeling so comfortable with disabling your anti-malware software. I'm going through the same thing and thankfully getting by with, "Waiting on an official fix from Microsoft". Not sure how long this will last without hearing from them but some random articles suggest Cyber attack and others like this thread suggest bad updates. I see everyone bypassing anti-malware and I'm still hesitant to even give that a whirl in an abundance of caution. Anyone else else on the same boat and can counter with a more concrete response besides what OP is saying. I do believe it's on the right track but why is disabling the anti-malware the first thing that comes to mind as a good workaround? Thanks! * permalink * embed * save * report * give award * reply [-]FST-LANE[S] 13 points14 points15 points 6 hours ago (1 child) Turns out it's not actually a bad update. It's a bug related to the maximum value of a signed integer. I explained all that in the original post near the bottom. Personally, I am not concerned with turning off the built-in anti malware component, because we have a third-party filtering solution in front of exchange which catches anything bad. * permalink * embed * save * parent * report * give award * reply [-]the_bushman924 5 points6 points7 points 6 hours ago (0 children) Thanks for creating this post! You've saved a lot of New Year hangover headaches! Agreed on the third party filtering that I have as a first layer of filtering defense. I feel more comfortable knowing this. * permalink * embed * save * parent * report * give award * reply [-]elint 8 points9 points10 points 6 hours ago (1 child) Why are you guys feeling so comfortable with disabling your anti-malware software. Because it's supplemental and hardly necessary. I've got a spam filter sitting in front of my Exchange server, filtering all inbound/ outbound mail. Honestly, it catches most malware before it ever gets to Exchange. Occasionally, when something slips through, more often than not it also slips through Exchange's malware scanner, and fortunately, my user training has been sufficient to keep users from clicking sketchy things. I'd be a lot more afraid of disabling it if it was my only protection, but then you've likely got other problems. * permalink * embed * save * parent * report * give award * reply [-]the_bushman924 1 point2 points3 points 6 hours ago (0 children) Thanks! Same here and good to know! It is more comforting that many of the replies are saying this. * permalink * embed * save * parent * report * give award * reply [-]nobody554Sr. Sysadmin 6 points7 points8 points 6 hours ago (1 child) In our case, we have other solutions in place that help scan for malware and such (external spam/malware filter). Microsoft's scanner would ideally never even see any bad mail because our first line of defense caught it all. That said, if you want mail to flow before Microsoft fixes their blunder, this is where you outweigh the risks of disabling one control vs keeping any other controls you have in place to protect your environment. * permalink * embed * save * parent * report * give award * reply [-]the_bushman924 1 point2 points3 points 6 hours ago (0 children) Thanks! Good to know. We do have external spam filtering as well. I will take that into consideration. * permalink * embed * save * parent * report * give award * reply [-]its_the_revolutionIT Manager 3 points4 points5 points 6 hours ago (0 children) We use other products from third party vendors like FireEye that focus on sanitizing mail before it arrives to Exchange. I'm confident this filter wouldn't find much of anything after it goes through those appliances we use. * permalink * embed * save * parent * report * give award * reply [-]Forgotmyaccount1979 8 points9 points10 points 19 hours ago (0 children) Oh man, I thought I was going crazy there for a minute. So glad I popped onto Reddit. * permalink * embed * save * report * give award * reply [-]ARDiver86 7 points8 points9 points 18 hours ago (1 child) Does this affect O365 or just conveniently on-prem instances? * permalink * embed * save * report * give award * reply [-]FST-LANE[S] 11 points12 points13 points 18 hours ago (0 children) So far, O365 seems to be fine for my tenants. But perhaps O365 has a delayed update mechanism or something. I wouldn't put it past Microsoft to push updates to on-prem Exchange first before they push the same updates to the O365 infrastructure. * permalink * embed * save * parent * report * give award * reply [-]CompetitionOk1582 8 points9 points10 points 15 hours ago (5 children) Why didn't this affect all on premise customers? * permalink * embed * save * report * give award * reply [-]jmch783 5 points6 points7 points 15 hours ago (3 children) I think the malware filter server update URLs are different for each part of the world. So those exchange servers "ahead" of time were impacted first and then once aware of the issue, MSFT took down the update URLs for those regions that hadn't been updated yet? That appears to have happened to my environment - servers in EU impacted but US based servers not impacted (yet anyway). * permalink * embed * save * parent * report * give award * reply [-]JeepMunkee 3 points4 points5 points 15 hours ago (0 children) My 2016 server in Colorado stopped processing at 545pm MST * permalink * embed * save * parent * report * give award * reply [-]172pilotsteve 1 point2 points3 points 8 hours ago (1 child) I suspect 3rd party malware agents disable the Microsoft ones. A friend supports a large installation that uses Mcafee and it was unaffected. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) load more comments (1 reply) [-]ADSWNJ 8 points9 points10 points 7 hours ago (1 child) Latest rumor I heard (just as a customer)... 1. "Y2K22" bug confirmed, as discussed in this thread. 2. Hotfix to be released asap to strip out the offending 2022-serial number sig files. 3. Temp workaround is to add a 13th month to 2021 for the signatures - i.e. the next sig after 211231nnnn will be 211301nnnn. So welcome to the 13th month of 2021! * permalink * embed * save * report * give award * reply [-]praetorfenixSysadmin 6 points7 points8 points 6 hours ago (0 children) Lousy Smarch weather! * permalink * embed * save * parent * report * give award * reply [-]SithLordDooku 8 points9 points10 points 5 hours ago (3 children) Update: My submission queue didn't clear because I had a transport rule that was still using the Filtering services. This is after I bypassed filtering and disabled it. I needed to disable the transport rule in order to get the email flowing. The event ID you are looking for is 4010. Transport engine failed to evaluate condition due to Filtering Service error. The rule is configured to ignore errors. Details: 'Organization: '' Message ID ' <2~~0220101132702.87329ce2ee2dc006@mail.com~~\>' Rule ID 'cd3d85a6-0c77-4d49-988b-88928b8a73aa' Predicate '' Action ''. FilteringServiceFailureException Error: Microsoft.Exchange.MessagingPolicies.Rules.FilteringServiceFailureException: FIPS text extraction failed with error: 'Exception of type 'Microsoft.Filtering.ScanAbortedException' was thrown.'. See inner exception for details ---> Microsoft.Filtering.ScanAbortedException: Exception of type 'Microsoft.Filtering.ScanAbortedException' was thrown. Get-transportrule -identify disable the transport rule. * permalink * embed * save * report * give award * reply [-]KingDoggles 2 points3 points4 points 4 hours ago (0 children) Thanks for this. This was necessary in our environment as well. I ended up disabling all of the rules for now (mostly audit stuff that is on by default). * permalink * embed * save * parent * report * give award * reply [-]jinzing 1 point2 points3 points 3 hours ago (0 children) Thank you very much you saved me a morning filled with dread. I couldn't for the life of me get my submission queue down until I read your post. Thank you so much about the transport rule tip. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]WaitHonest4926 7 points8 points9 points 15 hours ago* (21 children) Since a couple of minutes Microsoft released Engine 1.1.1880.4 and Sig. 1.355.1224.0 which is working like a charm. MS Filtering Engine Update process has successfully committed and handed off updates for MicrosoftLast Checked:2022-01-01T08:30:23ZLast Updated:2022-01-01T08:30:39ZEngine Version:1.1.18800.4Signature Version:"1.355.1224.0"Update Version:2201010004Last Definition Update:?2022?-?01?-?01T01:03:32.000ZUpdate Path:http:// amupdatedl.microsoft.com/server/amupdate Cheers and happy new year Chris * permalink * embed * save * report * give award * reply [-]xrtnn 2 points3 points4 points 14 hours ago (1 child) MS Filtering Engine update isn't resolving for me still getting The FIP-FS "Microsoft" Scan Engine failed to load. PID: 15996, Error Code: 0x80004005. Error Description: Can't convert "2201010003" to long. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]JudeCPer 1 point2 points3 points 14 hours ago (7 children) Updated with that signature, still fails. Error Description: Can't convert "2201010004" to long * permalink * embed * save * parent * report * give award * reply load more comments (7 replies) [-]DogResponsible8491 1 point2 points3 points 12 hours ago (3 children) process has successfully committed I'm still getting the FIPFS error after this update. The FIP-FS "Microsoft" Scan Engine failed to load. PID: 17860, Error Code: 0x80004005. Error Description: Can't convert "2201010005" to long. * permalink * embed * save * parent * report * give award * reply load more comments (3 replies) load more comments (7 replies) [-]Mottster 8 points9 points10 points 2 hours ago (1 child) Here is information from the Exchange Team Blog: https://techcommunity.microsoft.com/t5/exchange-team-blog/ email-stuck-in-transport-queues/ba-p/3049447 * permalink * embed * save * report * give award * reply load more comments (1 reply) [-]absoludicrous 5 points6 points7 points 19 hours ago (0 children) I saw the entries in Event Viewer and figured this couldn't have affected only us. Disabling the filter helped get mail flowing again. Thank you all for this thread. Anyone have any links from Microsoft? Maybe next year? * permalink * embed * save * report * give award * reply [-]briskik 7 points8 points9 points 16 hours ago (1 child) You guys are the best. Still took me 2 hours worth of troubleshooting to realize to come here * permalink * embed * save * report * give award * reply [-]ThreshOP 2 points3 points4 points 16 hours ago (0 children) Yepp same here. Thought for sure it was something dumb in my environment, Google wasn't showing any results yet, sysadmin saves once again. * permalink * embed * save * parent * report * give award * reply [-]reddi-tom 18 points19 points20 points 13 hours ago* (2 children) After troubleshooting over 1.5 hours, opened Reddit and of course you guys have a workaround. Gotta love Reddit r/sysadmin GMT+1 BTW, mailflow stopped around 2AM * permalink * embed * save * report * give award * reply [-]hakan_loob44I do computery type stuff 2 points3 points4 points 11 hours ago (0 children) 6am here. Noticed that I didn't have any email from onprem since last night. Figured that wasn't right. Spent 20min poking around then I figured that this had to be something MS fucked up and didn't post yet. That's when I came right to r/sysadmin. Reddit saves the day again. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [+]NeverEv3rGiveUpWindows Admin 4 points5 points6 points 12 hours ago (0 children) Just received a call from MS as a response for a seberity "A" case. They confirmed this approach as a good workaround and are working on the fix. * permalink * embed * save * report * give award * reply [-]happiAdmin 3 points4 points5 points 17 hours ago (0 children) Tnx guys, command and Transport service restart worked for me as well. Sigh. * permalink * embed * save * report * give award * reply [-]Impossible_Rush_2722 4 points5 points6 points 16 hours ago (0 children) Dude, love you right now. Though I called my sysadmin to help us do this for a few servers, woke him up, he might hate you.... But I love you! * permalink * embed * save * report * give award * reply [-]Justicefruitpies 4 points5 points6 points 4 hours ago (0 children) I love you all! Seriously, whoever jumped on this saved a day with my family. * permalink * embed * save * report * give award * reply [-]172pilotsteve 4 points5 points6 points 4 hours ago (3 children) FYI - Somewhere in these comments was someone having problems with mail delays even after disabling the malware filter as everyone is doing. I was having the same problem, with mail delays up to 45 minutes with thousands of messages in queue even while they were [slowly] flowing. I can report that after disabling (unchecking) my transport rules under the mail flow tab, my mail is flowing fast again. Hopefully this will help someone else who may also have transport rules. * permalink * embed * save * report * give award * reply [-]torbar203whatever 2 points3 points4 points 3 hours ago (0 children) https://www.reddit.com/r/sysadmin/comments/rt91z6/ exchange_2019_antimalware_bad_update/hqtt5ib/ More info on how to find the correct transport rule * permalink * embed * save * parent * report * give award * reply [-]jbrumsey 1 point2 points3 points 3 hours ago (1 child) I was having the same issue. Didn't have to disable all of my rules, only the ones looking at message attachments. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]freemantech757 3 points4 points5 points 4 hours ago (0 children) Really saved myself and my team a huge headache. Many thanks for bringing this light and to the entire reddit community for coming together to work through it when many of us have yet to hear a word from Microsoft. If I could give all the awards I would! * permalink * embed * save * report * give award * reply [-]Prancer_TruckstickSysadmin 10 points11 points12 points 18 hours ago (4 children) Jesus Christ, losing my mind around 9 PM trying to figure out why our queues weren't clearing... Disabling malware filtering did the trick for now. * permalink * embed * save * report * give award * reply [-]MarkDePalma 7 points8 points9 points 18 hours ago (3 children) The real answer is to rollback to the previous engine version and disable updates. That is what I am looking into. Transport rules can still be affected by this issue depending on if the rules need to invoke FIPS. * permalink * embed * save * parent * report * give award * reply [-]Prancer_TruckstickSysadmin 5 points6 points7 points 18 hours ago (2 children) That's a good point, keep oneself protected somewhat until a correctly formed definition becomes available. If you do find a way to roll back the definitions and disable updates, let me know, I'd appreciate it. * permalink * embed * save * parent * report * give award * reply [-]MarkDePalma 7 points8 points9 points 18 hours ago (0 children) As soon as I figure out a way I'll post a little write-up. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]PublicEntertainer 10 points11 points12 points 17 hours ago (0 children) u/FST-LANE Thank you so much for posting this. You saved me from working most of the night!! * permalink * embed * save * report * give award * reply [-]its_the_revolutionIT Manager 2 points3 points4 points 19 hours ago (0 children) Just got called for this as well, wtf MS! 2016 here. * permalink * embed * save * report * give award * reply [-]justingscu 2 points3 points4 points 18 hours ago (0 children) Yep just ruined my night almost :) I had to bypass malware engine and stop it's service and restart transport service to get email to flow (all 3 servers had the same error!!) * permalink * embed * save * report * give award * reply [-]maxnor1 3 points4 points5 points 15 hours ago (0 children) Thank you for sharing this! I'm glad it happend on the 31th, so my hope isn't destroyed that 2022 all will be better. * permalink * embed * save * report * give award * reply [-]Hasslemoffz 4 points5 points6 points 8 hours ago (0 children) As usual, when Microsoft fucks us up, the community steps in. Thanks for the thread guys, saved me a nasty few hours debugging this morning. * permalink * embed * save * report * give award * reply [-]techblackops 2 points3 points4 points 6 hours ago (9 children) Disabling antimalware solved the mailflow issue for me, but I've got a separate (appears related) issue with the exchange admin center login. I get the error below. Certs are good and our main cert was just renewed about a month ago. HMACProvider.GetCertificates:protectionCertificates.Length<1 Any ideas? * permalink * embed * save * report * give award * reply [-]Livid-Lie4603 4 points5 points6 points 6 hours ago (6 children) I have had that issue before and used this article to solve it. https://docs.microsoft.com/en-us/exchange/troubleshoot/administration /cannot-access-owa-or-ecp-if-oauth-expired * permalink * embed * save * parent * report * give award * reply load more comments (6 replies) load more comments (2 replies) [-]mypuppysunny 3 points4 points5 points 6 hours ago (0 children) Thank you for everyone that posted on here and the post author. If it weren't for this community I would feel pretty alone and screwed sometimes. * permalink * embed * save * report * give award * reply [-]rswwalker 4 points5 points6 points 5 hours ago (2 children) That's just fabulously bad programming using a signed variable to store unsigned data. Did they have interns writing code for Exchange server? * permalink * embed * save * report * give award * reply [-]elint 2 points3 points4 points 4 hours ago (1 child) That's just fabulously bad programming using a signed variable to store unsigned data. Personally, I don't think a version string like this should be stored as a signed/unsigned variable at all. It's a concatenation of several numbers and doesn't really require any sort of math. Store it as a string. * permalink * embed * save * parent * report * give award * reply [-]rswwalker 1 point2 points3 points 3 hours ago (0 children) Using an int or string is a design decision, but using a signed int to store unsigned data is just incompetence. * permalink * embed * save * parent * report * give award * reply [-]ecar13 2 points3 points4 points 3 hours ago (0 children) Whoever the fuck predicted Y2K was off by 22 years. * permalink * embed * save * report * give award * reply [-]Sad-Butterscotch5919 3 points4 points5 points 3 hours ago (1 child) If disabling malware transport agent does not fix it for you, look for 4010 events. Copy the transport rule ID and run a get-transportrule -identity . Disable that transport rule that is shown. (Seems to happen with any rule that scans attachments.) Restart transport service on all servers. * permalink * embed * save * report * give award * reply load more comments (1 reply) [-]B5565 3 points4 points5 points 3 hours ago (0 children) I was able to get my team on this and mail flow back up before midnight. Overall, we got it going before anyone actually noticed. Thanks r/Sysadmin ! * permalink * embed * save * report * give award * reply [-]Wasteway 6 points7 points8 points 15 hours ago (0 children) What a freaking joke. Come on MSFT, what in the actual hell. Much thanks to all of those who found this first and posted on how to fix it. You saved the rest of us tons of time! * permalink * embed * save * report * give award * reply [-]Steven20221978 2 points3 points4 points 16 hours ago (0 children) Great work. Happened just as we were making global firewall changes so lots of finger pointing this saved me * permalink * embed * save * report * give award * reply [-]RiceeeChrispiesJack of All Trades 2 points3 points4 points 10 hours ago* (3 children) Updated filtering engine signature to 1.355.1234.0 (released this morning) but email stops flowing when re-enabling Antimalware scanning and restarting transport service - still with the same error. Anyone experiencing this? UK based. Edit: Reading into it, guess we've got to wait from the boffins at MS. Sounds like just a signature update won't fix.. * permalink * embed * save * report * give award * reply [-]disclosure5 6 points7 points8 points 10 hours ago (2 children) I sure hope that "fix" doesn't look like a three hour Cumulative Update. * permalink * embed * save * parent * report * give award * reply [-]RiceeeChrispiesJack of All Trades 3 points4 points5 points 10 hours ago (1 child) Oh, it's Microsoft Exchange - you can almost guarantee it will be. Because every important security update requires you to basically reinstall Exchange, 'tis the only way. Boy, I really hope somebody got fired for that blunder. * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]SevereMiel 2 points3 points4 points 8 hours ago (0 children) we are still on exchange 2013 and had the same problem a week ago, when i've patched the server ... disabled the antimalware filter and probleme was solved * permalink * embed * save * report * give award * reply [-]MusicWallaby 2 points3 points4 points 7 hours ago (3 children) How can it be nearly 12 hours later and nothing at all anywhere from Microsoft? * permalink * embed * save * report * give award * reply [-]praetorfenixSysadmin 3 points4 points5 points 6 hours ago (0 children) How else they gonna up those O365 numbers? * permalink * embed * save * parent * report * give award * reply [-]rottenrealm 2 points3 points4 points 6 hours ago (0 children) kinda 'let those onprem freaks cry for a while" * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]dribar 2 points3 points4 points 7 hours ago (0 children) Fixed my 2019 environment. Much appreciated. Any news from MS about a patch? * permalink * embed * save * report * give award * reply [-]vikinickDevOps 2 points3 points4 points 5 hours ago (0 children) Why is it storing a version number as a LONG AND NOT A STRING? It makes sense if it was doing like an increment count 1, 2, 3 but anything more complicated than that just toss it in a goddamn string. I'm actually rationally mad at the engineering choices made by the developers. * permalink * embed * save * report * give award * reply [-]Mister_Big_Stuff 2 points3 points4 points 4 hours ago (0 children) Thanks for posting about this. Saved my morning by giving me the solution for a crazy issue that would have taken me a long time to figure out myself. * permalink * embed * save * report * give award * reply [-]handlebartender 2 points3 points4 points 4 hours ago (1 child) I've got nothing of value to add here. Just amazed. Random: although this is shite for users, I imagine there might be a few people over at MS who went from "enjoying the New Year" to "all hands on deck". * permalink * embed * save * report * give award * reply [-]FST-LANE[S] 2 points3 points4 points 4 hours ago (0 children) ...or at least we hope! Lol * permalink * embed * save * parent * report * give award * reply [-]boardhoarder 2 points3 points4 points 3 hours ago (0 children) Thank you to everyone for keeping me sane here! * permalink * embed * save * report * give award * reply [-]tourneytalk24 2 points3 points4 points 2 hours ago (0 children) I am not sure if it has been posted but if you have any mail flow rules that check attachments, you will want to disable those as well. The rules checking for attachments seem to use a common mechanism. We initially disabled the anti-malware scanning but still had backed up queues until finally discovering the rule was also an issue. Happy New Year! * permalink * embed * save * report * give award * reply [-]MarkDePalma 2 points3 points4 points 1 hour ago (0 children) I just created a script and wrote an article on rolling back the FIP-FS engine version (and temporarily disabling updates). This will resolve any residual email delays due to transport rules that use the FIP-FS engine. It also enables you to re-enable all malware filtering. https://blog.markdepalma.com/?p=810 * permalink * embed * save * report * give award * reply [-]chillyhellion 9 points10 points11 points 19 hours ago (1 child) Microsoft and broken updates. Name a more iconic duo. * permalink * embed * save * report * give award * reply load more comments (1 reply) [-]PublicEntertainer 3 points4 points5 points 17 hours ago (0 children) Thanks so much for sharing this. Microsoft killed my New Years Eve! * permalink * embed * save * report * give award * reply [-]DogResponsible8491 4 points5 points6 points 16 hours ago (0 children) Yes, currently been up half the night trying to find info on this issue. 'The FIP-FS "Microsoft" Scan Engine failed to load. PID: 12128, Error Code: 0x80004005. Error Description: Can't convert "2201010002" to long.' Got to love Microshaft. * permalink * embed * save * report * give award * reply [-]diezeldeez_ 2 points3 points4 points 9 hours ago* (0 children) Because of this post we did not wake up to a barrage of calls in email shit storm. Thank you, very much for posting this. Edit: people will downvote anything, I was just saying thank you. * permalink * embed * save * report * give award * reply [-]pssssn 1 point2 points3 points 19 hours ago (0 children) Also encountering this, thank you for the post. * permalink * embed * save * report * give award * reply [-]itguy3001 1 point2 points3 points 18 hours ago (0 children) Saved my ass. Thanks! * permalink * embed * save * report * give award * reply [-]NeverEv3rGiveUpWindows Admin 1 point2 points3 points 18 hours ago* (0 children) Happy new year! Same thing here guys! We have more than 150 our customers affected. * permalink * embed * save * report * give award * reply [-]Young-G0ku 1 point2 points3 points 18 hours ago (1 child) Wish I found this before spending an hour scratching my head, thanks Microsoft haha. * permalink * embed * save * report * give award * reply [-]praetorfenixSysadmin 1 point2 points3 points 6 hours ago (0 children) Did same thing. Came to Reddit to see if I was nuts. * permalink * embed * save * parent * report * give award * reply [-]togenshiJack of All Trades 1 point2 points3 points 18 hours ago (0 children) Yep affected as well, bypass malware filter until further notice. * permalink * embed * save * report * give award * reply [-]-----0----- 1 point2 points3 points 17 hours ago (9 children) Are these updates pulled from Windows Update or some other process. I'm on vacation but wondering if I should warn someone. * permalink * embed * save * report * give award * reply [-]its_the_revolutionIT Manager 3 points4 points5 points 17 hours ago (4 children) No, Exchange does it and you should definitely warn someone if you are using Exchange on-premise * permalink * embed * save * parent * report * give award * reply [-]FST-LANE[S] 3 points4 points5 points 17 hours ago (2 children) It's probably already broken and they're trying to figure it out without having to bother you while on vacation. Lol. * permalink * embed * save * parent * report * give award * reply [-]-----0----- 1 point2 points3 points 17 hours ago (0 children) Thanks. * permalink * embed * save * parent * report * give award * reply [-]-----0----- 1 point2 points3 points 16 hours ago (0 children) Yup it was broken. They ran the command and it's working again. Thanks for the info! Happy new year! * permalink * embed * save * parent * report * give award * reply load more comments (1 reply) [-]BrFrancis 3 points4 points5 points 16 hours ago (3 children) You're on vacation. Enjoy the fact that they can't even email you about it * permalink * embed * save * parent * report * give award * reply load more comments (3 replies) [-]BuschLightDrinkn 1 point2 points3 points 16 hours ago (0 children) Wow. Thank you so much for posting this!! Confirmed PS script worked for us. Great job finding this. * permalink * embed * save * report * give award * reply [-]DogResponsible8491 1 point2 points3 points 16 hours ago (0 children) I've disabled FIPFS, restarted the transport service, bypassed filtering and *still* getting the errors showing and email not being sent. Transport engine failed to evaluate condition due to Filtering Service error. The rule is configured to ignore errors. Details: 'Organization: '' Message ID ' [KTKATYL4PFU4.39SHENK9W21W3@BY1PEPF00001B88] (mailto:KTKATYL4PFU4.39SHENK9W21W3@BY1PEPF00001B88)' Rule ID '845cc901-be66-401c-9e22-deb0ab244ec1' Predicate 'containsDataClassification' Action ''. FilteringServiceFailureException Error: Microsoft.Exchange.MessagingPolicies.Rules.FilteringServiceFailureException: FIPS data classification failed with error: 'Scan request timed out on the queue:'. See inner exception for details ---> Microsoft.Filtering.ScanQueueTimeoutException: Scan request timed out on the queue: at Microsoft.Filtering.InteropUtils.ThrowPostScanErrorAsFilteringException (WSM_ReturnCode code, String message) at Microsoft.Filtering.FilteringService.EndScan(IAsyncResult ar) at Microsoft.Filtering.FipsDataStreamFilteringService.EndScan (IAsyncResult ar) at Microsoft.Exchange.MessagingPolicies.Rules.FipsFilteringServiceInvoker.ScanComplete (IFipsDataStreamFilteringService filteringService, ScanCompleteCallback scanCompleteCallback, IAsyncResult asyncResult, ITracer tracer) --- End of inner exception stack trace --- at Microsoft.Exchange.MessagingPolicies.Rules.FipsFilteringServiceInvoker.GetDataClassifications (Dictionary`2 classificationsToLookFor, FilteringServiceInvokerRequest filteringServiceInvokerRequest, ITracer tracer, FilteringResults& textExtractionResults) at Microsoft.Exchange.MessagingPolicies.Rules.BaseTransportRulesEvaluationContext.get_DataClassifications () at Microsoft.Exchange.MessagingPolicies.Rules.TransportRulesEvaluationContext.get_DataClassifications () at Microsoft.Exchange.MessagingPolicies.Rules.MessageProperty.OnGetValue (RulesEvaluationContext baseContext) at Microsoft.Exchange.MessagingPolicies.Rules.Property.GetValue (RulesEvaluationContext context) at Microsoft.Exchange.MessagingPolicies.Rules.ContainsDataClassificationPredicate.OnEvaluate (RulesEvaluationContext baseContext) at Microsoft.Exchange.MessagingPolicies.Rules.PredicateCondition.Evaluate (RulesEvaluationContext context) at Microsoft.Exchange.MessagingPolicies.Rules.AndCondition.Evaluate (RulesEvaluationContext context) at Microsoft.Exchange.MessagingPolicies.Rules.RulesEvaluator.EvaluateCondition (Condition condition, RulesEvaluationContext evaluationContext) at Microsoft.Exchange.MessagingPolicies.Rules.TransportRulesEvaluator.EvaluateCondition (Condition condition, RulesEvaluationContext evaluationContext). Message-Id:[KTKATYL4PFU4.39SHENK9W21W3@BY1PEPF00001B88] (mailto:KTKATYL4PFU4.39SHENK9W21W3@BY1PEPF00001B88)' * permalink * embed * save * report * give award * reply [-]SuitAdministrative96 1 point2 points3 points 16 hours ago (0 children) Our Exchange 2016 was affected. Thanks for this! PS script worked like a charm. Hope MS fixes this soon. * permalink * embed * save * report * give award * reply [-]insufficient_fundsWindows Admin 1 point2 points3 points 15 hours ago (0 children) Thank you jesus! I spent from 11:30-12:30 looking into reports of mail not going out; fortunately for my org, we're entirely on o365, but just have internal servers to act as mail relay for stuff generated by on-prem systems... but still - on just one of our 10ish exchange boxes, i saw 400+ mails queued by 1a. I haven't personally dealt with Exchange issues in literally years... once every 10 weeks on-call, and we almost never get exchange related issues.. geez * permalink * embed * save * report * give award * reply [-]jmch783 1 point2 points3 points 15 hours ago (0 children) Does anyone know if Microsoft has already pulled this update from the URL listed in the primaryupdatepath? For example, our EU based nodes were impacted but US-East nodes appear to be unaffected (as of now). The EU based primaryupdatepath URL is different from the US-East URL. * permalink * embed * save * report * give award * reply [-]wirtnix_wolf 1 point2 points3 points 14 hours ago (0 children) hi, that post saved my day! Thank you folks! is there information about when MS will send correct updates again so the malwarefilter can be activated again? * permalink * embed * save * report * give award * reply [-]l337scum 1 point2 points3 points 14 hours ago (0 children) Disabling Antimalware scanning in the transport agent is the only way to get past this right now. Been following it here: https:// twitter.com/JRoosen/status/1477120097747677184 * permalink * embed * save * report * give award * reply [-]praetorfenixSysadmin 1 point2 points3 points 11 hours ago (0 children) Holy crap you saved my still half drunk bacon! * permalink * embed * save * report * give award * reply [-]carfo 1 point2 points3 points 7 hours ago (0 children) fuck i learned my lesson. tried to troubleshoot this for hours this morning and of course it's fucking MS. kept asking myself: what changed? sigh. well thanks reddit * permalink * embed * save * report * give award * reply [-]PizzaCatLover 1 point2 points3 points 7 hours ago (0 children) Thanks for the post. We're unaffected because we're on 365, but we have clients who are impacted. Good to know what's going on. Y2K came 22 years late! * permalink * embed * save * report * give award * reply [+]Fun_Fan_9641 1 point2 points3 points 7 hours ago (0 children) Yep wasted a good part of the morning on this. Thanks Microsoft! * permalink * embed * save * report * give award * reply [-]bsitko 1 point2 points3 points 7 hours ago* (0 children) Another Microsoft fail in a year full of them. I can confirm the bug on my onprem 2016. I can also confirm that this workaround works. * permalink * embed * save * report * give award * reply [-]Swampycore 1 point2 points3 points 6 hours ago (0 children) Thanks for the info! Restored mail flow before anyone even noticed. * permalink * embed * save * report * give award * reply [-]ShakesTech 1 point2 points3 points 6 hours ago (0 children) Thanks so much disabling worked for me. * permalink * embed * save * report * give award * reply [-]The_Great_88 1 point2 points3 points 6 hours ago (0 children) Thank OP! Our 9 OnPrem 2016 CU22 all required this to get mail following * permalink * embed * save * report * give award * reply [-]carp3tguySysadmin 1 point2 points3 points 6 hours ago (0 children) Went and checked my Exchange queue after seeing this post and sure enough I'm affected, thanks for posting buddy * permalink * embed * save * report * give award * reply [-]DdraigJack of All Trades 1 point2 points3 points 5 hours ago (0 children) Should have checked this 14 hours ago :D * permalink * embed * save * report * give award * reply [-]carpetflyer 1 point2 points3 points 5 hours ago (0 children) Thank you! You saved me. I thought we hit back pressure and I was increasing disk space of the root drive. Still wasn't working. * permalink * embed * save * report * give award * reply [-]Al3nMicL 1 point2 points3 points 5 hours ago (0 children) I guess being a sys-admin really is a 24/7, 365 day kind of job. Lol * permalink * embed * save * report * give award * reply [-]rs-sysa077 1 point2 points3 points 4 hours ago (0 children) Thanks, nice way to wake up this morning. Appreciate your post. We have mail flowing again and await Microsoft's fix. Good grief. * permalink * embed * save * report * give award * reply [-]BalzovaiSysadmin 1 point2 points3 points 4 hours ago (0 children) Thanks crew, I was able to implement the fix almost immediately! You guys were spot on! Like others, I'll check back tonight and see if the MS defs/engine are patched. West Coast US here so I was last to the 'game'. 8) * permalink * embed * save * report * give award * reply [-]YetAnother_pseudonymExchange Admin 1 point2 points3 points 3 hours ago (2 children) I got called by our offshore support team at 4:30 AM CST for this. I was too tired to check my normal sources of info for what was going on and ended up opening a SEV A with Microsoft, and shortly after that got an email from them telling me to disable the malware transport agent, which worked after also restarting the transport service. We had submission queues of 20K + on a bunch of servers, but they cleared up pretty quickly. * permalink * embed * save * report * give award * reply load more comments (2 replies) [-]Theoneandonlyzeke 1 point2 points3 points 3 hours ago (0 children) Affects 2013 also as it happened to two of our servers already * permalink * embed * save * report * give award * reply [-]rob-entre 1 point2 points3 points 2 hours ago (1 child) The article states that 2013 is affected, but the three clients I have with Exchange 2013 were not affected. They could process mail normally. For what it's worth, I don't think the malware filter has done much of anything on my Exchange deployments, as a standard practice is to put a good filter in front of Exchange. While nice to have the additional scanners on Exchange, you should never depend on them alone. * permalink * embed * save * report * give award * reply load more comments (1 reply) [-]Tom_Neverwinter 1 point2 points3 points 2 hours ago (0 children) I was literally 10 minutes from leaving when this hit. FML * permalink * embed * save * report * give award * reply [-]nizmozSysadmin 1 point2 points3 points 1 hour ago (1 child) Thank you. We had the same issue today and I came across this article that provided the fix! We bypassed ours for now to get smtp working again. Figures MS would release something with a bug. Please keep us posted on an update to fix it and how. Thanks again! * permalink * embed * save * report * give award * reply [-]anachronous_one 1 point2 points3 points 1 hour ago (0 children) Much appreciated. This thread is getting enough recognition that it's being referenced in Mimecast service alerts as "the place" for their partners to go for information about this issue. * permalink * embed * save * parent * report * give award * reply [-]pascalbraxalt.binaries 1 point2 points3 points 1 hour ago (0 children) work called me on the first day of the year because all mails stopped, thank you reddit for figuring out why. * permalink * embed * save * report * give award * reply [-]JazDotKiwi 3 points4 points5 points 15 hours ago* (0 children) Thanks team from Sysadmin, we've sorted out our customers that have local Exchange servers. Does anyone know if there is impact to 365 services? Edit: I've done some basic testing with our 365 customers but so far seems to be ok. * permalink * embed * save * report * give award * reply [-]Glittering-Term-3583 1 point2 points3 points 14 hours ago (0 children) Microsoft have had such a poor year or so with security (netlogon, no-auth exchange RCEs come to mind) and other issues like this. * permalink * embed * save * report * give award * reply [-]cbiggersCaptain of Buckets 1 point2 points3 points 13 hours ago (0 children) Idiots. Who needs sleep or relaxation right? * permalink * embed * save * report * give award * reply [-]beeri0 1 point2 points3 points 11 hours ago (0 children) Thanks guys u really saved my butt. Microsoft seems to really hate us onprem costumer.. Maybe 2022 will gift us some nice onprem zerodays. Guys be careful and always update ur exchange as soon as possible. U never know. Happy New Year to all Exchange peps out there. * permalink * embed * save * report * give award * reply [-]apie77 1 point2 points3 points 9 hours ago (0 children) You are a legend. Just saved me hours of troubleshooting AND made me look very competent at the same time :) * permalink * embed * save * report * give award * reply [-]sidneymacsid 1 point2 points3 points 9 hours ago (1 child) Thank you Microsoft yet again for f**king up our systems with your BS untested updates. At least you had the decency to update it over a public holiday so admins could work on fixing what you broke with limited interruption to the business; it's not like we have a life anyway, we are here to serve you, our Lord and Master. Bill Gates, you have a lot to answer for...I can think of one word to describe you and it rhymes with anchor. * permalink * embed * save * report * give award * reply load more comments (1 reply) [-]Bleakbrux 1 point2 points3 points 3 hours ago (0 children) Luckily we are full hybrid. We only use our on premises box for smtp relay on Alerts, Firewall event's etc. I thought Today was quiet! * permalink * embed * save * report * give award * reply load more comments (88 replies) * about * blog * about * advertising * careers * help * site rules * Reddit help center * reddiquette * mod guidelines * contact us * apps & tools * Reddit for iPhone * Reddit for Android * mobile website * <3 * reddit premium * reddit coins * redditgifts Use of this site constitutes acceptance of our User Agreement and Privacy Policy. (c) 2022 reddit inc. All rights reserved. REDDIT and the ALIEN Logo are registered trademarks of reddit inc. Advertise - technology [pixel] p Rendered by PID 215634 on reddit-service-r2-loggedout-574b65f5df-th9n6 at 2022-01-01 23:00:52.315121+00:00 running f65e55f country code: US.