[HN Gopher] Tell HN: By default, New Relic will start collecting...
___________________________________________________________________
Tell HN: By default, New Relic will start collecting production log
data on 5/3
Unless you've been very careful, your production logs almost
certainly contain secrets or personally identifying information. I
was surprised (and annoyed) to receive the email below from New
Relic, stating that on 5/3 they will start ingesting all production
log data, by default. To make matters worse, if you provisioned
New Relic through Heroku, you can only opt out (by enabling High
Security Mode) if you contact support. And if you're on the free
plan, you can't open support tickets so have to ask on the
community forum. > New Relic APM agents will collect log data
starting 5/3! We've been hard at work making improvements to our
APM and logging capabilities, and when you update certain APM
agents starting May 3, 2022, logs will be automatically collected
and sent to New Relic One. Logs are a critical telemetry type for
observability and this new feature will help you troubleshoot your
applications faster. You probably have a few questions--including
how to customize your logs ingest--so we're including a FAQ below.
FAQ: Q: Why did you make this change? A: Logs are a critical
telemetry data type but they are messy. This improvement allows
users to send contextualized log data to New Relic without any
additional setup. Relevant log data is now surfaced and correlated
with other application telemetry automatically, reducing the need
to switch context or run log queries when troubleshooting your
applications. Q: Which APM agents will have automatic logs
collection and ingest upon upgrade? A: Starting May 3, 2022, when
users upgrade to the latest version of the Java, Ruby, and .NET
agent, log ingest will be enabled by default, unless High Security
Mode is enabled or you have enabled the logs toggle for your
accounts (more information on this below). We expect to enable
application logs for Node.js, Python, and Go by July and PHP by
September. Q: I have already implemented logs in context. What
should I do? A: We recommend only using manual OR automatic log
forwarding. For more information, check out this documentation. Q:
I already use a third-party log forwarder, or forward logs via the
New Relic infrastructure agent. What should I do? A: To avoid
duplicating log data, consult this documentation. Q: What does
this mean for my New Relic bill? A: Collecting application logs
means that more data will be ingested into the platform, at your
standard ingest rate. The APM agent samples logs to ensure optimal
agent performance. You can increase or decrease an application's
log volume as desired. Learn more here. Q: I am concerned about
sensitive log data being sent to New Relic. What should I do? A: No
logs of any kind will be collected if High Security Mode is enabled
on the agent, even after the agent is upgraded. If you do want to
use New Relic Logs, it is also possible to configure drop filters
to ensure sensitive data is not stored in New Relic. If you have
not enabled High Security Mode, but still do not want to send logs
to New Relic, see the next question. Q: I do not want New Relic to
collect or ingest logs, even after I upgrade my agents. What should
I do? A: You can either configure the agent config file locally on
a machine to disable it, or you can disable logs ingest for APM
agents at the account level with a toggle in the New Relic data
management hub. The toggle can be flipped before ever setting up an
APM agent that forwards log data. Q: Where can I learn more? A:
Check out our documentation, read the Explorer's Hub post, reach
out to your account team, or contact New Relic Support.
Author : ed
Score : 168 points
Date : 2022-04-28 17:33 UTC (5 hours ago)
| darkwater wrote:
| I was going to post "this must be a thing only in the USA, no way
| they are doing this to European customers due to GDPR", then I
| went to check my work email and bam, there it is. Gonna set the
| environment variables to disable this first thing in the morning
| tomorrow.
| aowen wrote:
| Ugh, I was not happy reading the email this morning. At least you
| can say they're being transparent with the FAQ, 'yeah, this will
| increase your bill'.
|
| We already had logging disabled, but changing the default
| behavior in their favor is shady.
|
| New Relic is already one of the most expensive items in our
| budget. Last year, I had to tell a producer "yeah, our SaaS that
| monitors our servers is more expensive than the servers
| themselves". That changed this year (started ramping up for
| scale), but it's still in the among the top of services we pay
| for.
| falcolas wrote:
| Reminds me of when they intentionally enabled JFR on every single
| java process. Roughly sextupled our ingest (and associated costs)
| until we got it shut back off.
| 29athrowaway wrote:
| I hope it becomes a legal nightmare for them and become the
| example of what not to do.
| AtNightWeCode wrote:
| New Relic, the company that tweaked MySQL to ingest more data
| than anybody could imagine. Cool. But New Relic is very obsolete
| in 2022 and should not be used.
| tapoxi wrote:
| Wow, we handle PHI and this is absolutely not acceptable. Glad
| we're not using New Relic but I'm terrified of Datadog doing this
| in the future.
| WkndTriathlete wrote:
| PHI must only be collected on platforms that are certified for
| PHI. Collection PHI on platforms not certified for PHI is a
| HIPAA violation and a good way to get sued for all of your
| dollars x10, as well as criminal proceedings.
|
| Fully-anonymized PHI can be analyzed on anything but the
| anonymization process must also be certified to have a
| reasonably low risk of re-identification when combined with
| other (arbitrary) datasets.
|
| (source: I used to implement and certify PHI data collection
| platforms.)
| dragonwriter wrote:
| > Collection PHI on platforms not certified for PHI is a
| HIPAA violation
|
| No, it's not. (There are compliance reauirements, both direct
| and implicit on the HITECH definition of secured PHI, but
| "certified for PHI" is not one of them, despite marketing
| myths spread by the private HIPAA certification industry,
| which pedals certifications without legal meaning.)
|
| > a good way to get sued for all of your dollars x10,
|
| HIPAA penalties are capped at a penalty of $1.5 million per
| year for violations of any single provision, regardless of
| number of violations (and $50k per violation).
| heavyset_go wrote:
| Most companies that deal with PHI are not covered entities,
| or their partners, that HIPAA restrictions apply to.
| draav wrote:
| I don't see how this would happen with Datadog. Their agent
| already has logging capabilities but you have to explicitly
| enable it
| aeyes wrote:
| NewRelic being predatory again, nothing new to see here. I'll let
| our account rep know that this makes us hate them even more.
|
| When they changed their billing to be per user our bill went up
| 20x for no additional value. Unfortunately I haven't found a
| better APM product that ticks all the boxes yet. I don't care
| about anything else but they don't understand that. And APM
| hasn't seen a single new feature in the 5 years I have used it.
| Stop taking my money to develop features I don't care about.
| aasasd wrote:
| > _I 'll let our account rep know that this makes us hate them
| even more._
|
| "Glad we could help! Just remember to continue making regular
| payments."
| bovermyer wrote:
| Yeah, APM is the only reason to use New Relic.
|
| There are other options out there, but surprisingly few open
| source ones. I know APM is hard, but...
| ksaxena wrote:
| Try this: https://github.com/SigNoz/signoz
| aeyes wrote:
| I checked the demo videos, it looks like it has the same
| problems I have with most NewRelic competitors: Too much
| focus on traces.
|
| In a high-throughput distributed system, traces start being
| mostly noise due to random latency spikes which are usually
| not very interesting.
|
| NewRelic is a lot more focused on the average but it breaks
| it down nicely so that you know where to start optimizing
| code. I rarely look at the p95 traces to do that.
| [deleted]
| alphabettsy wrote:
| This sounds only slightly less irritating than the Datadog agent
| which has features on by default that are billed at the on-demand
| rate.
| xvello wrote:
| > features on by default that are billed at the on-demand rate.
|
| Can you please elaborate on what these features are? I used to
| work in the agent team and we were cautious not to have that
| happen.
| bradly wrote:
| You will be charged for the increase data usage from New Relic
| unless you opt-out.
| jacquesm wrote:
| For New Relic's sake I hope they have thought through the
| consequences under the GDPR because they may well be causing
| their customers to violate their data processing agreements or
| even the law and may cause end users' data to be processed
| without consent. The fines for such tricks are nothing to sneeze
| at and companies working hard to stay compliant would do well to
| stay away from a supplier that does not have their best interests
| at heart.
| sam0x17 wrote:
| New Feature: "The delete everything button -- it deletes
| everything! Great for GDPR requests!"
| jacquesm wrote:
| I think that's called 'close your New Relic account'.
| falcolas wrote:
| Pretty much. They stink at doing targeted deletions. It
| required manual deletion of data in batches the last time
| we needed it.
| iameli wrote:
| We (Livepeer) switched to self-hosted Loki recently for this and
| couldn't be happier. Completely S3-backed, cheapest option
| possible, adjusting your retention is as easy as setting an
| expiration rule on the bucket. Queries are as fast as the
| hardware you feed it 'cause it's just all your nodes doing a
| brute-force search.
|
| Query language is a bit of a PITA compared to some of the other
| options, but you get used to it.
| coredog64 wrote:
| Setting the following envvars will disable it:
|
| NEW_RELIC_APPLICATION_LOGGING_ENABLED=false
| NEW_RELIC_APPLICATION_LOGGING_FORWARDING_ENABLED=false
| phist_mcgee wrote:
| Why won't everyone just use fully qualified dates (2022-05-03)!
| For a second I thought the OP had the date in march and was
| confused.
| datalopers wrote:
| If you're wondering why, the answer is NewRelic ($NEWR) has a
| market cap of $4B while DataDog ($DDOG) has a market cap of $40B.
| altdataseller wrote:
| Datadog isn't exactly the best example of friendly pricing as
| well.
| speedgoose wrote:
| A bit strange to have this feature enabled after an upgrade. The
| following statement is now wrong:
|
| > New Relic services are designed to receive and process
| telemetry data on the performance of applications, systems, and
| infrastructure, which typically do not contain any personal data.
| Customers generally send very little additional personal data to
| our platform.
|
| https://newrelic.com/blog/how-to-relic/how-demise-of-privacy...
|
| On the topic of privacy and GDPR, I find the previous link a good
| summary of the arguments used by American companies that cannot
| respect GDPR. Microsoft or Google say the same things in a lot
| more words.
| msla wrote:
| I'd like to see someone take this analysis on. Is it valid? If
| not, why not, assuming American law?
|
| https://web.archive.org/web/20200813235643/http://slawsonand...
|
| > Article 3(2), a new feature of the GDPR, creates
| extraterritorial jurisdiction over companies that have nothing
| but an internet presence in the EU and offer goods or services
| to EU residents[1]. While the GDPR requires these companies[2]
| to follow its data processing rules, it leaves the question of
| enforcement unanswered. Regulations that cannot be enforced do
| little to protect the personal data of EU citizens.
|
| > This article discusses how U.S. law affects the enforcement
| of Article 3(2). In reality, enforcing the GDPR on U.S.
| companies may be almost impossible. First, the U.S. prohibits
| enforcing of foreign-country fines. Thus, the EU enforcement
| power of fines for noncompliance is negligible. Second,
| enforcing the GDPR through the designated representative can be
| easily circumvented. Finally, a private lawsuit brought by in
| the EU may be impossible to enforce under U.S. law.
|
| [snip]
|
| > Currently, there is a hole in the GDPR wall that protects
| European Union personal data. Even with extraterritorial
| jurisdiction over U.S. companies with only an internet presence
| in the EU, the GDPR gives little in the way of tools to enforce
| it. Fines from supervisory authorities would be stopped by the
| prohibition on enforcing foreign fines. The company can evade
| enforcement through a representative simply by not designating
| one. Finally, private actions may be stalled on issues of
| personal jurisdiction. If a U.S. company completely disregards
| the GDPR while targeting customers in the EU, it can use the
| personal data of EU citizens without much fear of the
| consequences. While the extraterritorial jurisdiction created
| by Article 3(2) may have seemed like a good way to solve the
| problem of foreign companies who do not have a physical
| presence in the EU, it turns out to be practically useless.
| nix0n wrote:
| Currently, a lot of US companies have significant financial
| presence in Ireland, for tax-loophole reasons. See [0] for
| the Microsoft example specifically.
|
| I'm not a lawyer, and I know nothing about New Relic
| specifically. But the worst offenders of the spirit of GDPR
| really are vulnerable to EU enforcement.
|
| [0] https://www.theguardian.com/world/2021/jun/03/microsoft-
| iris...
| msla wrote:
| Right, I fully understand that, and it's an important fact.
|
| I'm talking about from the perspective of a small US
| company with no assets outside the US.
| whakim wrote:
| This may already be implicit in your question, but that
| putative company has to be clearly intending to offer
| goods or services to EU visitors (which isn't
| specifically defined but might mean something like
| accepting payment in Euros or advertising towards EU
| citizens). This seems like a vanishingly small percentage
| of "small US companies" although I do see your point re:
| enforcement.
| mhitza wrote:
| If you don't respect the GDPR, don't have a
| representative/establishment in EU/EEA... I haven't heard
| of anything like this happening before, but an EU judge
| could sanction your company. And Stripe, Paddle, PayPal,
| or any other payment service provider would abide to.
|
| Likelyhood of that happening? 1/1mil? Who knows. EU GDPR
| enforcement is only getting better, and the scenario you
| have asked about is sure to be in the spotlight sooner or
| later.
| msla wrote:
| > If you don't respect the GDPR, don't have a
| representative/establishment in EU/EEA... I haven't heard
| of anything like this happening before, but an EU judge
| could sanction your company. And Stripe, Paddle, PayPal,
| or any other payment service provider would abide to.
|
| That's interesting: Sanctioning you by going after
| business partners. That could be messy, and could spark
| laws about that here were a European court to actually
| try it. How much power _do_ we want foreign courts to
| have over our citizens?
| jacquesm wrote:
| The easiest way is then to simply not track your EU
| customers. That takes care of 99% of the problem.
| msla wrote:
| I'm interested in enforcement mechanisms a European court
| would have over a purely American company that did track
| European customers. People around here seem to assume
| that Europe has enforcement powers over the entire world,
| which seems bizarre to me. Do we also collectively assume
| North Korea can enforce its laws in Silicon Valley?
| jacquesm wrote:
| America seems to believe the exact same thing. The
| typical way is that they pluck execs out of airport
| lines.
|
| For instance:
|
| https://en.wikipedia.org/wiki/David_Carruthers
|
| Was a pretty high profile case like that.
|
| You may of course disagree with that but I think that if
| you intend to do business in a country that you should
| abide by the laws of that country, even if there is no
| direct way to enforce them. Sooner or later you may find
| that nation states have pretty long reach.
| jiveturkey wrote:
| If you are logging Personal Data, is that even wise, wrt GDPR?
| Logs have a funny way of escaping, NEWR notwithstanding. I'm
| not a Privacy Engineer, but surely everyone understands that
| you don't log raw passwords. Similarly, if you are bound by
| GDPR, shouldn't you not log PD?
| mhitza wrote:
| From my experience most "shops" don't disable IP address
| logging in their load balancers/web servers, which is
| personal data under the GDPR and California privacy act.
| jiveturkey wrote:
| very good point!
| speedgoose wrote:
| You shouldn't log personal data but you may do it without
| planning to do so. For example you may log some error
| messages that contain personal data.
| [deleted]
| Nicksil wrote:
| >Q: What does this mean for my New Relic bill? A: Collecting
| application logs means that more data will be ingested into the
| platform, at your standard ingest rate. The APM agent samples
| logs to ensure optimal agent performance. You can increase or
| decrease an application's log volume as desired. Learn more here.
|
| Automatic, opt-out bill increases come 5/3?
| postpawl wrote:
| They did something similar with distributed tracing. Seems like
| dirty business tactics are becoming the trend at newrelic.
| msla wrote:
| "We're going to start charging you to grab up your PII unless
| you explicitly tell us to stop. YOU'RE WELCOME!" sounds like it
| ought to be illegal _somewhere_ the New Relic developers have
| to worry about the laws of.
|
| I wonder what they'd do if someone rigged their install to feed
| their servers junk information. Is that possible? I honestly
| don't know how the New Relic stuff works.
| nickstinemates wrote:
| You can feed their servers as much arbitrary shit as you want
| - you'll just end up paying for it.
| jacquesm wrote:
| It is illegal. The question is who will be liable, NR or the
| customer, at first glance it probably will be the customer
| even though NR is the one causing the violation. Regardless,
| this kind of move should be punished by customers cancelling
| en masse.
| rootusrootus wrote:
| Ah, so that's how they're paying for the huge number of jobs
| they seem to be trying to fill all the time. Some days my
| entire job feed is just New Relic.
| wincy wrote:
| Curious where do you look for jobs? I'm in the mortgage
| industry and might be a good time to start looking.
| rootusrootus wrote:
| I have a running LinkedIn search for 'principal software
| engineer' and 'software engineer' in Portland that has been
| going for a while. I'm not really looking right now, but I
| like to see what the trends look like.
| ge96 wrote:
| Curious what does that mean? "running" it's
| automated/bought or you just search time to time?
| grumple wrote:
| If you do a job search in linkedin, there's an option at
| the top left to set an alert.
| rootusrootus wrote:
| Yep, it's automated. LinkedIn has the concept of saved
| searches, and these are mine. It sends me an email every
| day or so with a list of matching jobs in the local area.
| lelandfe wrote:
| I just landed a job after 8 months of applications. Oof.
| The usefulness probably varies city to city, but for NY
| tech jobs I got the most mileage out of Built In[1]. Being
| able to filter with a checkbox for "frontend jobs" was
| nice. I was also a daily user of Indeed (awful) and
| LinkedIn (saturated with bad jobs - but their alerts are
| lovely!).
|
| All three of these sites have a large amount of "promoted"
| jobs that show up. I'd recommend filtering them out with a
| script or user style.
|
| I wasn't looking for a start up role, but have heard good
| things about AngelList[2] for those that are.
|
| [1] https://builtin.com/tech-hubs
|
| [2] https://angel.co/
| smallerfish wrote:
| New Relic's logging agent had a bug in it that lost a good
| portion of our logs for a few months last year. Support were
| extremely lethargic about diagnosing, and once they found that it
| was in fact a bug on their end, weren't at all apologetic. We
| dropped them and moved to DD for logging and it has been
| generally better so far, although I know by reputation they have
| price surprises now and again.
|
| Amusingly, despite our account being closed, I received this same
| email this morning, so somebody there isn't exactly on top of
| keeping their mail list clean.
| hn_version_0023 wrote:
| IME, they will never stop sending you email.
| hnlmorg wrote:
| DD pricing is a f^<king mess for larger organisations. I've
| spent more energy figuring out DD bills than I have AWS. And it
| doesn't help that DD Account Managers are just as clueless.
| mvf4z7 wrote:
| Now might be a good time to check out Dynatrace if you are a
| current New Relic customer. https://www.dynatrace.com/
| bovermyer wrote:
| Do they have a free tier, or are they "enterprise only?"
| ksaxena wrote:
| Or go open-source: https://github.com/SigNoz/signoz
| sofixa wrote:
| That's weird. They don't bill for logs per GB/line, iirc, so from
| a financial perspective it seems an odd choice - it will cost
| them lots of money to just pile on lots of logs from
| users/companies who didn't ask for it ( if they needed them they
| would have enabled the feature themselves).
| ibz wrote:
| How if they are planning to mine and sell data?
| magundu wrote:
| I don't think the will do it.
|
| The reason may be logs can contribute to lot of ingestion
| volume compared APM, RUM, Infra products. They make lot of
| money in this case.
| beberlei wrote:
| their current pricing works based on ingested data in GB. I
| suppose logs falls under that.
| krono wrote:
| Any recent changes to their data processing terms that could
| explain this?
|
| Just thinking out loud, I'm personally not familiar with this
| service.
___________________________________________________________________
(page generated 2022-04-28 23:00 UTC)