[HN Gopher] Accused murderer wins right to check source code of ...
___________________________________________________________________
Accused murderer wins right to check source code of DNA testing kit
Author : anfilt
Score : 799 points
Date : 2021-02-07 09:33 UTC (1 days ago)
(HTM) web link (www.theregister.com)
(TXT) w3m dump (www.theregister.com)
| hastradamus wrote:
| The co-founder of the company, Mark Perlin, is said to have
| argued against source code analysis by claiming that the program,
| consisting of 170,000 lines of MATLAB code, is so dense it would
| take eight and a half years to review at a rate of ten lines an
| hour.
|
| This is hilarious. As if you need to read every damn line and you
| can't skip blank lines? You can skip whole files that aren't
| relevant. Weak excuse
| jariel wrote:
| The statement is odd, at the same time, it's not outrageous for
| him to make in the sense that - lines of code notwithstanding -
| the underlying science i.e. the application of the product is
| the thing in question.
|
| It's a pretty interesting case.
|
| At least the core nature of the algorithm should be made public
| if we're going to use it for public inquisition.
| heavyset_go wrote:
| "It could take too long :(" is an interesting excuse not to
| examine the evidence that could put someone behind bars for the
| rest of their life.
| zyngaro wrote:
| This kind of software should be required by law to be open
| source.
| TheRealDunkirk wrote:
| Yes, just like voting machines. But how do we confirm that the
| software which was vetted is the software that was actually
| _used_?
| mseidl wrote:
| Nobody should use electronic voting.
| Spivak wrote:
| People keep saying that but electronic voting would be
| great! Set it up so the process is _exactly_ like mail-in
| ballots except digital and you can't have made it any
| worse. It's not like emailing a PDF to be opened by an
| election official is any different than mailing a document
| to be opened by that same official.
| Ekaros wrote:
| No one should use mail in voting either...
| Spivak wrote:
| Except for the sick, disabled, elderly, deceased, people
| overseas or out-of-town, anyone in the military that's
| deployed, people who don't have good access to
| transportation, people who work long continuous shifts
| like healthcare workers and firefighters, poll workers,
| people in jail awaiting their trials, people with
| stalkers...
| ryukafalz wrote:
| How many Nigerian princes have asked for your bank
| account number via physical mail? How about via email?
|
| The internet lowers the barrier to certain forms of abuse
| substantially enough that I don't think you can so easily
| say you won't have made it any worse.
| Spivak wrote:
| I mean none because Gmail's spam filters are pretty good
| these days but I take your point. I'm not at all saying
| people should be trained to vote via unsolicited email.
| You would have to ask for digital voting when you
| register. And you would of course be able to check that
| your ballot was received through the same online form
| that already exists for mail-in ballots.
|
| Right now you already give election officials your email
| for mail-in ballots and if there's a problem with your
| ballot they'll email you!
|
| I think we could do a lot better than this when it comes
| to online voting but as a baseline optionally replacing
| the transport from physical mail to email I don't think
| would be the end of the world.
| [deleted]
| heavyset_go wrote:
| Well, for voting, you just don't use machines. Counting by
| hand is an easy algorithm to audit, and literally any adult
| that can count is able to audit the process themselves.
| calciphus wrote:
| Compile it, run it with the same inputs the prosecution
| claims was provided, and see if you get the same result.
| That's sorta the point of computers, right? Same inputs on
| the same program, same output?
| glitchc wrote:
| Same compiler? Same configuration? Same OS? There will be
| variations in the output. How much variation is reasonable?
| fsflover wrote:
| https://reproducible-builds.org/
| [deleted]
| curryst wrote:
| > Compile it, run it with the same inputs the prosecution
| claims was provided, and see if you get the same result.
| That's sorta the point of computers, right? Same inputs on
| the same program, same output?
|
| One difficulty here is that the input is a real world
| effect. That means the answers are rarely exactly the same.
| The last time I looked, breathalyzers have a ~50% margin of
| error. So if you blow a 0.06, your BAC is really somewhere
| between 0.03 and 0.09. You could mod your breathalyzer to
| always assume that the reading was 25% higher than the
| reality without being noticed, as long as your breathalyzer
| doesn't return a result that's already 25+% higher during
| testing. Even if it does, your other results will be in
| range.
|
| They need to just stop using breathalyzers. They're
| inaccurate pieces of machinery, operated by people without
| any medical knowledge to speak of, who have a vested
| interest in the readings being high. The whole thing, from
| top to bottom, is just a long chain of poorly aligned
| incentives. Field sobriety tests are even worse. The fact
| that your license can be revoked for refusing the test
| under the above grounds is nothing short of an end-run
| around the 4th amendment by declaring driving a "privilege,
| not a right" despite the fact that it is extremely hard to
| live in most of the US and not be able to drive. How the
| fuck do I survive in rural
| Oklahoma/Georgia/Kentucky/Texas/etc, 40 miles from the
| nearest bus stop, without a car? And all without any due
| process. It's never impacted me, but it bothers me
| nonetheless.
|
| They need to force people to go in for a blood draw to
| establish BAC. It is an accurate test, administered by
| competent medical professionals, who have no vested
| interest in the outcome either way. Yes it does take
| longer. Yes, some people who were at 0.08 will be down to
| 0.07 by the time the test is administered. I still find
| that more comforting than the fact that some people at 0.06
| will read at 0.09, and some people at 0.14 are going to
| read at 0.07.
|
| There's also the fact that a cop who doesn't calibrate
| their breathalyzer is merely going to get a slap on the
| wrist, and a medical professional who does the same is
| probably going to end up with a huge malpractice suit and
| possibly lose their license if it was egregious in some
| way.
| kart23 wrote:
| >The last time I looked, breathalyzers have a ~50% margin
| of error
|
| Source on this? I know the portable ones are like this,
| but I thought the actual court-admissible ones are more
| accurate, and the calibration procedures need to be
| followed to get a conviction. Also, FSTs are quite good
| for preliminary screening, a proper HGN test especially.
| And if you ask for a blood test, they are required to
| give you one and consider it in court.
|
| Cops have no incentive to make people who aren't drinking
| take FSTs or the breathalyzer, its just a waste of their
| time. They don't want to be bothering normal people, they
| want to be taking real dangerous and irresponsible
| drivers off the road. You survive by not drinking before
| driving, or having a DD, its really not that hard.
|
| Implied consent is the reason for the tests, and you
| agree to it when you sign for a license. And I think its
| a worthwhile tradeoff if it actually works to reduce DUI
| deaths.
| tsimionescu wrote:
| > [Cops] don't want to be bothering normal people, they
| want to be taking real dangerous and irresponsible
| drivers off the road.
|
| That's the nice theory we all wish we lived in, but the
| reality all over the world is that police will extremely
| often do things for their own petty reasons, often
| systematically. Police departments often have quotas for
| tickets, they can often extract a bribe by threatening
| someone with a DUI, they can simply want to scare
| 'undesirables' out of a 'nice neighborhood'.
|
| In general, there are good reasons to be extremely weary
| of the police and their motives.
| JoeAltmaier wrote:
| Not true universally. In-car breathalyzers are calibrated
| regularly; have margins much closer to 10%. They are
| portable and simple to use.
|
| Maybe in the 'bad old days' it was worse. But today its
| been scrutinized in court so many times, it has to be
| bulletproof to get deployed.
|
| Now, blood testing may differ from breath testing. Which
| is aligned with impaired driving? Breath testing may be
| the better measure. If state laws are often couched in
| terms of breath alcohol, then they are the only correct
| measure.
| themaninthedark wrote:
| > The whole thing, from top to bottom, is just a long
| chain of poorly aligned incentives. Field sobriety tests
| are even worse. The fact that your license can be revoked
| for refusing the test under the above grounds is nothing
| short of an end-run around the 4th amendment by declaring
| driving a "privilege, not a right"
|
| I agree and I wish I had a better solution.
|
| >They need to force people to go in for a blood draw to
| establish BAC.
|
| I know the intention is good, but that seems like a very
| scary proposition to me. I think the same poorly aligned
| incentives will line up here too.
|
| "We are now partnered exclusively with X-clinic to do all
| the blood tests."
|
| X-clinic just happens to tweak their procedure so that
| borderline samples end up showing under influence.
| Jolter wrote:
| Many jurisdictions would have enough cases come through
| to justify employing someone trained to draw a blood
| sample.
|
| Btw here (Sweden) the breathalyzer is only used to help
| decide whether someone gets picked up for a blood draw or
| not. Only a blood test can be used to convict someone.
| themaninthedark wrote:
| To be honest, having someone affiliated with the justice
| department would not be seen as something that increases
| trust.
|
| There are historical and current issues that cause this
| distrust, having an adversarial or multiple third parties
| is probably the best option for the US.
| antidocker wrote:
| What if compiler hijacked? What if the OS itself hijacked?
| What if the hardware has built in backdoor?
| [deleted]
| razakel wrote:
| It's staggering that a slot machine in Vegas has more
| official scrutiny than a voting machine.
| CivBase wrote:
| At some point you just have to trust people to not
| intentionally lie in a trial. We have very stiff penalties
| for anyone caught lying in court, but there isn't much more
| we can realistically do.
| jsjsbdkj wrote:
| We have stiff penalties for _civilians_ lying in court.
| Cops can lie in court, get caught, and still have the court
| accept their version of events as basis for proscution.
| pacamara619 wrote:
| It's not just like voting machines if it's just software.
| Just give it the same DNA as input and check if your results
| match the others.
| fsflover wrote:
| https://publiccode.eu
| duluca wrote:
| Things are only going to get murkier when leveraging ML. All of
| this needs to become open source.
| fritzo wrote:
| Agreed, and with ML we'll need both open source and open
| training data.
| known wrote:
| Sounds rational since conviction should be on irrefutable
| evidence;
| kazinator wrote:
| > _Mark Perlin, is said to have argued against source code
| analysis by claiming that the program, consisting of 170,000
| lines of MATLAB code, is so dense it would take eight and a half
| years to review at a rate of ten lines an hour_
|
| What an deplorably dishonest argument to present to laypeople.
|
| If there is some weakness in it so the accused has a way to
| plausibly deny the DNA test, it will take a consultant who has
| relevant competence something like week or two to find it.
|
| Finding a problem doesn't require looking at every line of
| 170,000.
| zupreme wrote:
| What is unfortunate is that it took going to appeal to force the
| judge to allow the code review at all.
|
| Without, at minimum, an independent review (and preferably open
| source code) the software and lab processes being used constitute
| an inscrutable "black box" process within which any judgment can
| be made, for any conceivable reason, with life-changing effects
| for the defendant (and for the victims of a crime if, for
| example, a rapist or murderer is set free by a non-match
| decision).
|
| One could even say that unreviewable code here falls under the
| umbrella of "secret evidence", which much of the world already
| knows can be easily misused and/or misapplied at the whim of the
| court.
| criddell wrote:
| In Canada you can't even get the breathalyzer maintenance
| records:
|
| https://www.canadianlawyermag.com/news/general/maintenance-r...
| vmception wrote:
| People sometimes ask me what my "number" is, like how much net
| worth or "money" I want, what would I do with it
|
| I say "I want to be able to afford appeals court where my
| rights matter"
|
| Infinite appeals court!
|
| Most people plea out, cant make bail, dont have counsel buddy
| buddy with the judge enough to get you bail, and lose the
| ability to keep good counsel for more and more motions and
| appeals
|
| I want that, there is almost no pride in American rights if you
| cant afford them. People tie their whole identity to a system
| they arent even part of
| roywiggins wrote:
| Appeals? Only ~3% of people charged even go to trial, the
| rest plead out.
|
| Just giving everyone a substantive right to trial would
| amount to a revolution.
| vmception wrote:
| Yeah. But just giving _myself_ that right first, forever.
| lotsofpulp wrote:
| I have the same line of reasoning when people talk about
| having enough to feel secure. Even simple civil legal matters
| cost in the tens of thousands of dollars easy.
|
| And the system works so that you're either rich enough to be
| able to defend yourself and the money spent doesn't affect
| you, you're poor enough that you have nothing to lose, or
| you're in the middle, busy trying to get from poor to rich,
| but you are vulnerable to losing it all because you don't
| have enough to protect it, but you have enough that it's
| worth for someone else to try and take it.
| cyberlurker wrote:
| Yes, that middle zone (that most of us probably live in) is
| terrifying. And it's why we get conned into so many
| different types of insurance.
|
| "One disaster and all that progress is gone."
| hutzlibu wrote:
| "Yes, that middle zone (that most of us probably live in)
| is terrifying"
|
| More terrifying than the bottom where you got nothing to
| loose? I doubt it. Otherwise, why be afraid of it?
| vmception wrote:
| I would say just because of the energy and sacrifices
| used
|
| At the bottom you don't have to pretend that the
| circumstances will improve, and there is some freedom
| associated with some approaches to that. Careers don't
| need to have continuity, I know many people in
| hospitality and service industry whose vacation policy is
| saving and quitting one restuarant, travelling, and
| getting another job at a different restuarant when they
| get back. Sure other approaches have lots of energy used
| on finding food and shelter that day, and service and
| hospitality work is not necessarily at the bottom, my
| post isn't about those approaches and dilemmas.
|
| People in the distinct category of "professional"
| careers, not my term, don't feel like they have that
| freedom to have any timegaps and are resigned to earning
| small periods of time off, and often times that is true.
| ineedasername wrote:
| Or just make the entire system based on a sort of "public
| defender" model. As it stands, a person accused of a crime
| and then found innocent has still been punished without even
| being found guilty due to enormous legal bills. It is a
| highly asymmetric power structure for anyone who isn't
| wealth: the prosecutors have massively more resources than
| the average person to call upon. Alternatively, when
| prosecuting the wealthy, that asymmetry is reversed, which
| might be equally problematic.
| undersuit wrote:
| Guess you should just start doing these crucial DNA tests
| against some sort of panel of tests instead of just one lab. It
| would be a shame for the quality of the code in your one test
| to convict an innocent or free the guilty.
| WaitWaitWha wrote:
| The tone of this thread is leaning to "forensics evil, government
| lackeys out to get the little guy".
|
| In the USA, all sides can call and rely on their own forensics.
| There is no government mandated & approved single-source-of-truth
| (with some minor exceptions).
|
| > Forensic labs and companies are expert witnesses with black box
| processes and the incentive to protect the authority of their
| profession.
|
| This statement is at minimum an extreme generalization. Forensic
| field is a very large field, with many government and private
| "forensicators".
|
| There are no "black box processes". The very word "forensic" is
| based on presenting in full view, front of the fact finders (jury
| & judges usually).
|
| How deep this gets dug into depends on the fact finders and
| attorneys.
|
| In all court cases the forensic examiner can be called to
| demonstrate with extreme nuance how they performed the
| procedures. Any tools used are can be requested to be examined,
| including software. Vendors that I have worked with _all_ have
| experts on staff specifically to appear in court and detail the
| inner workings of their tools.
|
| Not only tools, processes, and environments, but the examiner can
| be drilled on their experience, education, degrees, previous
| cases, failures, etc. They do call it "voir dire" for a reason.
|
| This does not negate the fact that there are some bad apples, bad
| prosecutors, and bad judges.
| Falling3 wrote:
| > In the USA, all sides can call and rely on their own
| forensics.
|
| ... to the extent their financial resources allow.
| eeZah7Ux wrote:
| ...which can be extremely different between parties leading
| to EXTREMELY biased outcomes.
| Falling3 wrote:
| Exactly.
| WaitWaitWha wrote:
| I think your way of writing it is the the best expressed.
| WaitWaitWha wrote:
| There are several comments regarding that if one cannot pay for
| an opposing forensic investigation, then forensics is a black
| box.
|
| Either you are moving the goal post, or I am to donnish.
|
| Let's agree that it is scientifically _not a black box_ , but
| some may not be able to pay for such service.
|
| There were suggestions of nationalizing, centralizing or
| governing forensics and just have one, unbiased working for the
| courts.
|
| This diametrically opposed to the problem pointed out by many
| where prosecutors will use specific labs because they return
| more positives. If the prosecution and the forensicator work
| for the same employers, how is that prevent further erosion of
| this problem?
| dingoegret wrote:
| That's because forensics has a sordid history of being
| overvalued in trial cases because of magic thinking like "it's
| science it must be the truth". When in reality a lot of it is
| quackery and science is the last thing that leads to a simple,
| quick and direct truth.
| 0xfffafaCrash wrote:
| Sorry but forensic "expertise" in the court room has always
| been laughable. In "full view of the judge and jury" means
| absolutely nothing when these people are completely
| scientifically illiterate and defer to the so called expert who
| has a financial incentive to help the prosecutor. There's an
| entire industry of people who make their livings by reliably
| testifying to the guilt of defendants and then being
| compensated for their "expertise." They just need to be paraded
| as experts and how would a layperson be expected to know any
| better? The quality of defense in the American justice system
| has everything to do with who can afford to pay for it. I could
| cite a hundred sources but here's a small sample...
|
| [0] https://www.sciencemag.org/news/2016/03/reversing-legacy-
| jun...
|
| [1] https://pubmed.ncbi.nlm.nih.gov/30447642/
|
| [2] https://theconversation.com/how-corruption-in-forensic-
| scien...
|
| [3] https://www.newyorker.com/magazine/2009/09/07/trial-by-fire
| inetknght wrote:
| > _There are no "black box processes". The very word "forensic"
| is based on presenting in full view, front of the fact finders
| (jury & judges usually)._
|
| You're wrong.
|
| When the defendant can't afford an expert witness then the
| defendant is literally shut out from being able to understand
| the technology involved.
|
| Good luck proving to a jury made up of non-technical people how
| thread timing problems cause math problems in DNA analysis
| without an expert witness.
|
| Good luck getting an expert witness to testify to that when the
| source code is hidden behind a black box "company trade
| secret".
| ncallaway wrote:
| > In the USA, all sides can call and rely on their own
| forensics
|
| Ah, not quite.
|
| More accurately: "all sides can call and rely on their own
| forensics _if they can afford it_"
|
| In this country we _say_ that everyone has the right to an
| attorney, but that doesn't extend to expert witnesses.
|
| This means forensic evidence becomes a _phenomenal_ tool for
| targeting the little guy, or the lower classes, while
| conveniently providing hooks for the more wealthy defendants to
| escape the system.
| tal8d wrote:
| > There are no "black box processes".
|
| Do you know how much of the fingerprint match process is left
| up to the judgement of the examiner? You can't get much more
| "black box" than another person's brain. Last time I checked
| (several years ago), tool mark analysis was still without
| objective foundation. Also... bitemark analysis - that was a
| thing.
| dhdhhdd wrote:
| My experience is the following testimony: "i put the following
| parameters to the following program and this was the result.".
|
| Potentially explainig why the given pareters were chosen.
| anomaloustho wrote:
| > Any tools used are can be requested to be examined, including
| software.
|
| Out of curiosity, if this is a common occurrence that is
| willfully obeyed by all vendors and parties involved. Could you
| shed some light on why this vendor is resisting?
| WaitWaitWha wrote:
| No, it is not common but does happen. There are several cases
| on drug testing, DNA, bite mark, and software that comes to
| mind. Scientifically well established processes, tools, etc.
| are rarely called up.
|
| Bite mark was a big deal in the industry because it turns
| out, it is not so unique and the methodologies developed were
| weak.
|
| My personal opinion why the are resisting? They are resisting
| because they are bottom dwellers. Forensicators whom I
| associate with, tool vendors, and I understand that we have
| to share knowledge. I dare say, we are the most open source
| scientific knowledge industry.
|
| This is not because of some altruistic reason, but because
| tomorrow they can be called on to explain.
| LatteLazy wrote:
| We had a case in the uk with accounting software. Dozens of
| people were convicted over decades because the software couldn't
| be wrong. Until eventually someone actually checked.
|
| https://www.bbc.co.uk/news/business-54384427
| pvitz wrote:
| I have worked with MATLAB code with 20,000 lines of code. Only
| over the past years, OOP and unit-testing has become properly
| available and usable. My guess is that this 170,000 lines are
| written in the old procedural way (also for performance reasons)
| and are full of bugs, also thanks to the lack of supporting
| tools.
|
| Most likely, this grew out of a research prototype that just
| worked too well to be reimplemented in a proper production
| environment.
| fatnoah wrote:
| I wonder if bug tracker and other reports would be part of
| discovery in this case.
| pvitz wrote:
| Equally interesting is in my opinion who should do the
| review. Mathworks' own consulting service is probably the
| best to do so, but I wonder if they would objectively work
| against one of their own customers.
| fatnoah wrote:
| Yeah, any technical expert in a trial concerns me. I was an
| alternate juror (meaning I had to sit through the trial but
| was not allowed to take part in any deliberations) in a
| trial that involved the testimony of a computer "expert".
| The expert's testimony was 100% true and appeared to
| definitely prove X to someone who knew nothing about the
| subject matter. It was things analogous to saying the
| system was secure because it had a security chip.
|
| There were 1,000,000 questions I wished had been asked.
| egberts wrote:
| Well, DNA matching is crapshoot and hazardous toward the
| innocents. We can still find unrelated folks with partial match
| by the virtue of segmentation.
|
| https://dna-explained.com/2017/01/19/concepts-segment-size-l...
| rudylee wrote:
| Why can't just they run the DNA test again in 2 or 3 other
| different software ? If all of them come back with the same
| result then we can assume that the first software is not buggy
| right ?
| handoflixue wrote:
| It could be that they're all copying the same Stack Exchange
| answer, or making the same basic mistake - If 50% of
| programmers do averages wrong, it's not ridiculous to claim
| that 3 in a row got it wrong. Tripling your costs and still
| having a 12.5% failure rate isn't great.
| giantg2 wrote:
| Now if this can become precedent for all evidence derived from
| automated systems. There have been some past battles about BAC
| machines in this regard.
| Mashimo wrote:
| Yay, free code review?
| Snoozus wrote:
| Why don't they just redo the analysis with a few other software
| packages?
| MaxBarraclough wrote:
| That would help with variance, but not with bias.
|
| Business incentives presumably reward all such companies for
| helping the police get convictions, so it doesn't seem a far-
| fetched concern.
| elliekelly wrote:
| The defendant has a right to "confront" their accuser. So when
| an algorithm is the one doing the accusing it seems only fair
| that the defendant should be able to understand how and why the
| algorithm arrived at that conclusion. Running the analysis
| through every software package available wouldn't satisfy the
| defendant's 6th amendment rights.
| carlmr wrote:
| It wouldn't satisfy them, but if one of the software packages
| gave a different answer they can't both be correct.
| elliekelly wrote:
| Or worse, they could give the same incorrect answer.
| jonnycomputer wrote:
| Who is "they"
| unnouinceput wrote:
| Probably "they" refer to prosecution. And yeah, why is
| prosecution going with that lab/software only? One reason is
| probably the DNA available for more testing is not enough /
| one testing could been done reliable with DNA taken from
| crime scene. Other reason might be behind the scenes
| incentives, which seems is the defense strategy here
| jlgaddis wrote:
| > _why is prosecution going with that lab /software only?_
|
| Presumably becquse that lab/software gave them the answer
| they were looking for.
|
| Why pay for even more lab/software testing when you've
| already got what you want (especially when it's a
| possibility that the additional testing will contradict the
| first)?
| vaduz wrote:
| Even if you had plenty of DNA material, going with multiple
| labs comes with a risk that one of them comes with a "no
| match" result - and that is something that is supposed to
| be disclosed to defense as potential exculpatory evidence -
| and is a nice source of reasonable doubt for a jury. It's
| something that is terribly inconvenient for the prosecution
| when someone has already been charged with the crime...
| amelius wrote:
| How does this work? They only have to find 1 bug and the entire
| case is dropped / postponed?
| carlmr wrote:
| I guess they have to show this bug changes the result.
|
| But if you look at e.g. the case against Toyota where they had
| expert witnesses from Carnegie Melon and NASA testifying that
| the code was a horrible mess, I think that was sufficient for a
| multi billion dollar fine. So I guess with the admission from
| the DNA company that their code is basically unreviewable,
| written in a language known for prototyping, if they get an
| expert witness to testify this it may be enough without showing
| how the software misbehaves exactly.
| LockAndLol wrote:
| For the Europeans among us, I would like to remind you that
| there's an initiative to opensource all publicly funded code
|
| https://publiccode.eu/
|
| If you care, add your signature and vote for a party during the
| next elections that considers opensource at least somewhat
| important.
| brutal_chaos_ wrote:
| Disclaimer, I am FOR seeing the source in these cases.
|
| Say the defense finds the software in a very, very troubling
| state. Could that be telling enough to not trust the outcomes of
| the software? Assuming the DNA parts were correct, but the rest
| is junk, this is where the plaintiff would bring in their expert
| to counter the arguments?
|
| Perhaps ^ is just a weak argument. I don't know how well versed
| courts are in these matters. (My guess is NOT that well versed
| because of the CFAA of the 80's up to Aaron Schwartz, and more).
|
| My legal background is Law and Order, so I have no idea what I'm
| talking about, just curious.
| johnnyfived wrote:
| How would a team of independent reviewers even go about examining
| source code of this scale? How can you possibly find bugs that
| aren't super obvious?
| twobitshifter wrote:
| The defense team only needs to provide the jury with a
| reasonable doubt. They don't have to prove that it's entirely
| wrong, only the existence of mistakes that call into question
| the overall accuracy. The founder has already made their
| argument for them by claiming that the entire codebase is
| impossible to review. "since a complete review is not possible,
| we reviewed a sample of x files and found y errors in the code.
| It is obvious that the developers has not adhered to the strict
| software code quality review standards that are necessary when
| dealing with life and death situations."
| pavel_lishin wrote:
| > _How would a team of independent reviewers even go about
| examining source code of this scale? How can you possibly find
| bugs that aren 't super obvious?_
|
| Something tells me that 170k of Matlab code is going to contain
| _plenty_ of obvious bugs.
| motohagiography wrote:
| It's an important development. Forensic labs and companies are
| expert witnesses with black box processes and the incentive to
| protect the authority of their profession. They are as likely to
| lie as any other witness. Perhaps even moreso.
|
| "Those arguing on behalf of the defense cited past problems with
| other genetic testing software such as STRmix and FST (Forensic
| Statistical Tool). Defense expert witnesses Mats Heimdahl and
| Jeanna Matthews, for example, said that STRmix had 13 coding
| errors that affected 60 criminal cases, errors not revealed until
| a source code review." "They also pointed out, as the appeals
| court ruling describes, how an FST source code review "uncovered
| that a 'secret function . . . was present in the software,
| tending to overestimate the likelihood of guilt.'"
|
| An analogous situation with an alcohol and drug testing lab
| caused a scandal in Canada and called into question 16,000 child
| protective services cases:
| https://en.wikipedia.org/wiki/Motherisk , and then at the same
| hospital(!), a forensic pathologist was giving fake prosecution
| evidence,
| https://en.wikipedia.org/wiki/Charles_Smith_(pathologist)
|
| Maybe there's just something about Toronto and compromised
| processes, but defense challenges to the integrity of automated
| systems looks like a growth field.
| IG_Semmelweiss wrote:
| i think we can all agree that black boxes are not desirable.
|
| But let's not jump the gun here, and assume ill-intent.... to
| explain something that can be attributed to negligence, or
| simple human error.
| anamexis wrote:
| Why shouldn't we assume ill-intent? The precedent and the
| incentives are there.
| IgorPartola wrote:
| When your negligence results in people wrongfully spending
| decades in prison I would argue that it ought to be criminal
| negligence. If your software does not produce results that
| are accurate, precise, and have confidence intervals
| included, how OK are you with people going to prison based on
| them?
| crazygringo wrote:
| The quality standards need to be set by the courts or by
| leglislation.
|
| No software is perfect. It's unreasonable for any bug to be
| considered criminal negligence, or else the entire software
| development profession would be in prison.
|
| Instead, software that can result in life or death or
| prison scenarios needs to have tested, documented, verified
| quality controls, just like we do for vehicles.
|
| But that means courts or the legislature have to decide
| what those standards are.
| caconym_ wrote:
| Yeah, there need to be consequences for this. The
| alternative is empowering people and corporations to
| destroy lives with impunity in pursuit of their own goals,
| as long as they can later claim it wasn't intentional.
|
| What really pisses me off is how we fail again and again to
| ask the question of whether these entities should have been
| doing what they were doing in the first place, if it even
| _can_ be done safely. Facebook, Equifax, police
| surveillance and misconduct, totally unaccountable
| "forensics" techniques--in all these cases where
| incalculable damage is done in totally predictable ways, we
| only address it reactively (if we address it at all) and
| completely fail to fix or even really consider the root
| cause(s). It's like we have this huge blind spot where we
| take as a given that established entities doing thing(s)
| just have a right to keep doing them, regardless of any
| other factors.
|
| In this case, forensics "experts" (essentially witch
| doctors) and fly-by-night black-box "labs" with unproven
| methodologies an undergrad intern might have developed in a
| week or two are institutionalized in our courts. This has
| happened _because_ there is zero oversight and zero
| accountability for their claims, and lives are destroyed as
| a result.
|
| It is insane not to pursue criminal charges for these
| indiscriminate arsonists of justice--if they can't do what
| they claim to do fairly and accurately, they should _not do
| it_. They don't have a right to "try" just because it will
| make them money, regardless of the consequences.
| tsimionescu wrote:
| A lot of people respond to this as if the fact that it
| would be nice if forensics worked justifies treating them
| as if they do. 'What do you want them to do, stop looking
| for fingerprints and DNA evidence?' Yes! Just like I
| don't want them to use 'lie detectors' or hire psychics.
| caconym_ wrote:
| Exactly! It's pure wishful thinking, and in a lot of
| cases those wishes are (at best) frighteningly amoral in
| context.
|
| It's like a hack to get around the whole "reasonable
| doubt" thing: just sequester all that pesky doubt in a
| black box so that people can't see it, and if anyone asks
| just play dumb.
| drstewart wrote:
| >I would argue that it ought to be criminal negligence
|
| I would like a thorough analysis of everything you've ever
| learned that led you to this conclusion, and if we find one
| mistake in your lifetime of learning then this is also
| criminal negligence on your part.
| [deleted]
| kspacewalk2 wrote:
| I agree that we shouldn't _assume_ ill intent, but we should
| recognize it is a very real possibility. Ill intent comes in
| when you realize the simple human error. Alone, in your
| office, poring over code or reviewing past files. Then you
| weigh the impact on your career and reputation of honestly
| fessing up, and put it off for a while, then a while longer.
| AS37 wrote:
| > a forensic pathologist was giving fake prosecution evidence
|
| The doctor handled thousands of cases in his career, and a
| well-funded inquiry found issues with less than 1% of them.
|
| Sure we'd like that to be 0%, and society should spend time
| thinking of how much they rely on complicated processes of
| reasoning, but that's a really good accuracy.
| sethammons wrote:
| If the error goes against innocent people, then it must be
| unacceptable. If the error sides with the "would-be"
| criminal, that is more acceptable as we, as a society, have
| decided that letting a few bad folks go is preferred to
| imprisoning innocents.
| 35fbe7d3d5b9 wrote:
| > as we, as a society, have decided that letting a few bad
| folks go is preferred to imprisoning innocents.
|
| We, as a society, like to _say this_ because it makes us
| feel good.
|
| But a critical analysis of our actions reveals that we
| don't believe it for a second. We are happy to write
| overbroad laws that allow most behaviors to be
| criminalized, over police marginalized communities, and
| place those who have been convicted by this flawed system
| in deeply dysfunctional prisons that may well violate their
| human rights.
| yarcob wrote:
| From the Wikipedia page
|
| > In June 2005, the Chief Coroner of Ontario ordered a review
| of 44 autopsies carried out by Smith. Thirteen of these cases
| had resulted in criminal charges and convictions. The report
| was released in April 2007, indicating that there were
| substantial problems with 20 of the autopsies.
|
| That sounds more like 45% to me.
|
| Also this bit from the article about a baby that allegedly
| had a skull fracture:
|
| > Later exhumation of the child and examination of the skull
| have shown that there was no skull fracture. It is thought
| Dr. Smith confused the normal gap between the baby's skull
| plates for an injury.
|
| Holy shit that is bad. Even as a lay person I know about
| these gaps.
| AnIdiotOnTheNet wrote:
| > Holy shit that is bad. Even as a lay person I know that
| there are gaps between skull plates.
|
| Sure, but can you tell them apart from a fracture caused by
| injury? Presumably someone knowledgeable in medicine should
| be able to, but maybe it is one of those things that isn't
| as obvious as we, the uneducated outsider, might think.
| R0b0t1 wrote:
| That's an extremely flawed argument. If you can't explain
| it to a layman then you don't understand it. If you can't
| tell whether it's a gap or a fracture -- you can't get a
| conviction.
|
| "It just looks like it" is not a valid reason.
| AnIdiotOnTheNet wrote:
| It is possible that they could have well supported their
| assessment with evidence and still been wrong.
| AS37 wrote:
| Interestingly there's an area of psychology called
| Naturalistic Decision Making which studies how experts
| make decisions that they _can 't_ explain. (Example: a
| firefighter may be able to pinpoint where a fire is
| before they enter a house and see it.)
| dathinab wrote:
| > there are gaps between skull plates.
|
| Between Baby skull plates, not adult ones ;-)
|
| They grow together at some point.
| yarcob wrote:
| I edited my post to make it clear I don't think skulls
| have gaps in general.
| pc86 wrote:
| You say that as if these were mistakes or unfortunate
| culminations of circumstance. The man committed fraud and
| sent innocent, grieving parents to prison by proxy. I don't
| think it's unreasonable to expect that to be 0.00% and punish
| those harshly who exceed that metric.
| badRNG wrote:
| If I'm a juror, and the evidence presented happens to have
| a false positive rate of 1%, that certainly meets my
| understanding of "a reasonable doubt."
| londons_explore wrote:
| I would need a _very very_ big pile of very clear cut
| evidence to find someone guilty of a crime where the
| likely outcome is a substantial prison time.
|
| It's all tradeoffs, but "his blood was found on the
| murder weapon" wouldn't be sufficient for me - some
| childhood enemy could easily have planted a bunch of
| forensic evidence next to a crime scene.
| michaelt wrote:
| Unfortunately, the vast majority of crimes don't produce
| evidence that rises to the level you call for.
|
| After all, if my next door neighbour barged into my house
| and beat me up, the only evidence would be my visible
| injuries and my statement that it was my neighbour.
|
| I'm not sure if I'd prefer a society where he would be
| convicted, or where he wouldn't.
| londons_explore wrote:
| Maybe your neighbour has a nicer car than you which makes
| you feel insecure so you decide to beat yourself up and
| go to the police saying your neighbour did it.
|
| Considering that possibility, I don't want to put your
| neighbour in prison.
| michaelt wrote:
| And presumably, if having been denied justice I pursue
| revenge instead, I can barge into his home and beat him
| up, as he did to me, and avoid punishment for the same
| reason?
|
| While I admire your devotion to Blackstone's Ratio, this
| doesn't seem like a recipe for stability and rule of law
| to me.
| Nightshaxx wrote:
| A murderer kills less than 1% of all the people they
| meet. Does that mean we should just accept their
| behavior?
| slickQ wrote:
| A murderer who kills 1% of the people he meets has
| certainly committed a crime. A person who has a 99%
| chance of being a murderer has not certainly committed a
| crime.
|
| That is depending on your threshold of certainty. 1% is
| not that high considering that, according to the OJJDP, 5
| milion people were arrested for serious charges in 2019
| so with a 1% false positive rate that would be 100,000
| people falsely imprisoned every year.
|
| It does not have to be and really can't be 0% but 1% is
| unreasonably high in my opinion. If it can't be helped
| then it can't be helped but that isn't necessarily the
| case with these devices.
| throwaway2245 wrote:
| Is it bad that I'd guess you would be excluded from a
| jury if you admitted this upfront?
| mschuster91 wrote:
| Additionally it could be made a requirement that, like in
| professional sports, two or three blood drafts are taken -
| one sample is analyzed and one is either kept until the
| case is closed or analyzed at a different lab if the
| evidence is contested.
| londons_explore wrote:
| This method doesn't protect against evidence tampering.
| Someone can just take a drop of that blood and smear it
| on the victims clothes. Now it doesn't matter how many
| labs analyze it, it's always going to come back a
| match...
| yarcob wrote:
| The wikipedia page is heartbreaking. Griefing parents sent
| to jail, their other children taken away and placed in
| foster care... So many lifes made miserable by one
| person...
| dghlsakjg wrote:
| The end result of that tiny little error rate was tens of
| millions in settlements and legal costs, families permanently
| separated by government mandate, and multiple people spending
| more than a decade in prison.
| m463 wrote:
| Imagine what an innocent man goes through.
|
| If I was innocent, I'd be 100% certain that the system was
| screwed up, and if DNA evidence "proved" it, I'd want to look
| at the code too.
|
| I remember reading about the husband who was falsely accused of
| murdering his wife, and it was very very sad.
|
| https://www.texasmonthly.com/politics/the-innocent-man-part-...
| JoeAltmaier wrote:
| I'm not so sure we should be allowing challenges to expert
| witness, with no evidence. That sounds like a rich source of
| abuse of the system, like a big company filing thousands of
| briefs to delay a case.
|
| It's important for experts to use validated, tested equipment.
| That's clear. And by the sound of it that is happening (stories
| of how code was tested and found wanting confirm that; stories
| of double-blind testing).
|
| Maybe what is needed is, an 'underwriters lab' certification
| for such devices. To forstal the inevitable gaming of the
| 'right to challenge'.
| jonhohle wrote:
| In most criminal complaints the defendant has far fewer
| resources than then the prosecution. The prosecution has a
| conflict of interest to get a conviction. Absolutely
| everything about their process needs to be above reproach,
| including their experts. A challenge to that process,
| approved by a judge, seems like a small consolation in a
| biased system.
| sixothree wrote:
| When I was a juror on a murder trial the one thing that was
| most clear was the disparity in resources.
|
| On one side were 6 impeccably dressed men in what I
| presumed were multi-thousand dollar suits. On the other
| side was single overwhelmed, overworked, and under-dressed
| defender who was steam-rolled at every sentence she spoke.
|
| I ended up in traffic court last year for a parking ticket
| that I can only think must have been stolen from my car
| during a festival. After spending just a few hours waiting
| my turn, it was pretty clear who exactly the police target
| for traffic violations. But this is a tangent.
| JumpCrisscross wrote:
| It seems reasonable to have a lower bar for challenging
| expert witnesses in criminal cases versus civil ones.
| tehwebguy wrote:
| Absolutely not! Please, if you want to know why, look into
| expert witness testimony as a whole, possibly can start with
| "burn pattern experts".
|
| Not to mention we have a constitutional right (6th amendment)
| to face and cross examine witnesses against us in the US, a
| black box should not void that right.
| yholio wrote:
| > I'm not so sure we should be allowing challenges to expert
| witness, with no evidence
|
| Any challenge is "allowed" in the sense that the parties can
| try to convince the judge there is a flaw.
|
| The question is one of burden of proof: should the expert
| prove they have a trustworthy result, or should the expert
| testimony stand on their credibility alone, and the defense
| scramble to poke holes into it armed with incomplete
| information.
|
| For a system that can put people into jail for life and that
| acts as a direct extension of state power, it's obvious the
| latter can have disastrous consequences. Effectively, any one
| who acquirers the "judicial expert" title can put people in
| jail at their whim.
| gamblor956 wrote:
| When I was a public defender I made my career by cross-
| examining expert witnesses. There are a dozen innocent people
| walking around free today because I challenged the expert
| despite going into the cross "with no evidence."
|
| What you call abuse, I call due process.
|
| (And your proposed solution is already part of the expert
| certification and questioning process in criminal trials, so
| it isn't a solution. Experts must testify that the equipment
| they used was calibrated/certified/etc based on the standards
| applicable to the field.)
| JoeAltmaier wrote:
| Testify does not equal independently certified. Also,
| double-blind testing is not really subject to faking. Its
| the standard of science.
|
| Responses to my comment are all around 'how it is now'
| which seems not to be very reliable. A responsible
| testing/certification process would address that.
| gamblor956 wrote:
| You've just moved the goal posts from someone who can be
| crossed in court to someone who can't, which is worse,
| because then you have now way of verifying that the
| "independent" certifier is actually independent or that
| the certification was properly done.
|
| You still need to allow the "independent certification"
| to be examined in court to validate both the independence
| and the certification. The means testimony on the
| applicable standards, and how the specific equipment used
| satisfied those standards. And right now, the expert
| witness already does the latter with respect to
| calibration, etc., for the equipment they used.
| JoeAltmaier wrote:
| The skepticism has to end somewhere. Else ever court case
| becomes an infinite regressing into doubting the
| foundations of everything.
|
| Of course self-certification is not the best. To bring up
| that straw man (as most of these posts do) is not
| advancing the conversation.
| tsimionescu wrote:
| That straw man is essentially the status quo at the
| moment...
| JoeAltmaier wrote:
| Reread the comment being responded to? It started with
| that, suggested an alternative. Anything helpful to add?
| salamanderman wrote:
| Grossly oversimplifying it, but it is what my lawyer uncles
| say, and is consistent with what I saw as a juror on a civil
| trial involving many topics I'm personally an expert in,
| expert witnesses are paid to lie on the stand. At the very
| least they are paid very very well (the amount they were paid
| was emphasized at the trial I was at) to state only the
| information consistent with what the person paying them
| wanted to them to say, regardless of whether they know
| information that would be more helpful to the other side.
| kspacewalk2 wrote:
| Of course this is the case, but rather than lying, I think
| it is only a consequence of selection bias. You don't see
| the experts rejected by the prosecution (or defence) for
| opinions that aren't helpful, because _of course_ they won
| 't be invited to testify.
| JoeAltmaier wrote:
| They answer any question put to them. The defense has to
| ask the right questions of course.
|
| I know expert witnesses. They are of impeccable character,
| and will seek to educate as well as they are allowed.
| tsimionescu wrote:
| You know _some_ expert witnesses. I doubt you have a good
| acquaintance of a significant percentage of all expert
| witnesses.
| AS37 wrote:
| > At the very least they are paid very very well... to
| state only the information consistent with what the person
| paying them wanted to them to say
|
| Anecdotally, if you replace 'paid' with 'incentivized' my
| understanding is the same.
|
| And here's a totally unrelated Wikipedia link:
| https://en.wikipedia.org/wiki/Michael_Baden
| josephorjoe wrote:
| defendants should always be able to challenge expert
| witnesses qualifications.
|
| and how could anyone on hacker news claim there is 'no
| evidence' that unreviewed software may do something other
| than what people claim it does? especially if it 170,000
| lines of matlab?
|
| from the article:
|
| > "Without scrutinizing its software's source code - a human-
| made set of instructions that may contain bugs, glitches, and
| defects - in the context of an adversarial system, no finding
| that it properly implements the underlying science could
| realistically be made," the ruling says.
| x0x0 wrote:
| Particularly when TrueAllele specializes in
|
| > _probabilistic genotyping_ [1]
|
| and
|
| > _solves mixed DNA samples, low template DNA and kinship
| problems_ [1]
|
| And the product sheet discusses
|
| > _models for PCR artifacts_ [2]
|
| This sure feels like they go way beyond typical dna match
| technology into maximally extracting information via
| statistics, so their methods definitely should be open for
| review.
|
| [1] https://www.cybgen.com/products/casework.shtml
|
| [2] https://www.cybgen.com/solutions/brochures/unique_featu
| res.p...
| GizmoSwan wrote:
| DNA testing should be double blind. I have no idea if it is.
|
| Sample contamination can be arranged before sent to the lab
| though.
| MaxBarraclough wrote:
| > Forensic labs and companies are expert witnesses with black
| box processes and the incentive to protect the authority of
| their profession.
|
| I imagine they're also incentivised to favour a result
| indicating guilt.
| anonAndOn wrote:
| > incentivised to favour a result indicating guilt
|
| Exhibit A: Annie Dookhan of the Massachusetts State Crime Lab
| who is suspected of falsifying _thousands_ of drug tests
|
| [0]https://www.cbsnews.com/news/massachusetts-lab-tech-
| arrested...
| HarryHirsch wrote:
| Annie Dookhan did about 3 years of jail, which is risible,
| considering that her cooking of evidence affected > 30000
| cases. No visible consequences for the supervisor, which is
| scandalous, considering that the had been carrying on for
| at least 4 years.
| tantalor wrote:
| Why? The risk to their reputation & livelihood for a false
| positive should temper that.
| dathinab wrote:
| Because they likely use metrics which highlight how many
| criminals where caught due to their system.
|
| On the other side a metric which says how many times _no
| conclusive evidence was found_ with DNA testing sounds more
| like a metric of failure (but is not, if you don 't have
| qualitative good enough evidence including "complete
| enough" DNA you should not make up results which simply
| don't exists, but then courts ruling without conclusive
| evidence isn't that rare and some would say even necessary
| to some degree. It's just that many believe DNA tests are
| somehow unfailable perfects ways to prove the presence of
| an specific person at a specific place, but they are often
| not).
| vkou wrote:
| When a DNA lab issues a false positive, its CEO doesn't go
| to prison, nor do they get hit with a charge-back for all
| the tests they've done in the past year.
|
| Their reputation doesn't matter as much as you think.
| Police procurement isn't based on the reputation of the
| vendor. It's based on whether or not the vendor delivers
| the results they want.
|
| In fact, government procurement in general, isn't based on
| the reputation of the vendor. This is by _design_.
| _jal wrote:
| Livelihood is frequently the reason why biases creep in.
| Independent labs tend to do better.
|
| Here's a look at two different Texas crime labs. Compare
| and contrast, and see if you can spot the differences.
|
| https://gritsforbreakfast.blogspot.com/2021/02/beyond-aid-
| to...
|
| It starts with one pledging to: aid law
| enforcement in the detection, suppression, and prosecution
| of crime
|
| whereas the other seeks to provide
| medical examiner and crime laboratory services of the
| highest quality in an unbiased manner with uncompromised
| integrity.
|
| and goes from there.
| acranox wrote:
| Maybe for some, but look what happened in Massachusetts.
| https://en.wikipedia.org/wiki/Annie_Dookhan
| MaxBarraclough wrote:
| The hands-off reputation-based approach isn't enough to
| regulate food and drugs, or aviation. I don't see why for-
| profit forensics companies should be trusted implicitly.
|
| Also related: we know that funding bias [0] is a real
| problem in science, despite that scientists' reputations
| should (ideally) motivate them to resist such biases.
|
| [0] https://en.wikipedia.org/wiki/Funding_bias
| kortilla wrote:
| No, the question is what is the incentive for lying and
| claiming guilty? Do prosecutors choose different labs?
| Ma8ee wrote:
| Of course prosecutors will favour expert witnesses that
| will help them get convictions. Why isn't that obvious?
| JoeAltmaier wrote:
| Sometimes the expert witness is in the department's own
| lab. It seems a tautology that they will favor
| conviction. Every officer bringing in evidence is begging
| "Give me something I can use to get this scumbag!" The
| pressure to fudge is enormous.
|
| A fairly simple step is to certify equipment
| independently, and to do blind testing (one sample is
| provided from evidence; another by an independent
| unrelated source). Of course this costs more, but
| anything else is effectively an experiment without a
| control.
| drdrey wrote:
| There are countries where the job of the prosecutor is to
| uncover the truth, not to convict someone at all costs:
|
| https://en.wikipedia.org/wiki/Inquisitorial_system
| Ma8ee wrote:
| I know. I live in such country. But even here it is
| better for the prosecutor's career with many convictions.
| jerf wrote:
| I doubt there's any system in the world where it's
| _officially_ the job of a prosecutor to "convict someone
| at all costs". It certainly isn't in the US.
| Nevertheless....
| 2112 wrote:
| Most of the very real dynamics that affect all of our
| lives at a policy-making / systemic level are never
| official. If you think the official stuff is a reflection
| of reality ... I don't know how to finish this sentence.
| Safe from it being "official", it is very much the job of
| a prosecutor in the US to convict someone at all costs.
| This is common knowledge.
| Aerroon wrote:
| The Soviet Union is a perfect example of this. They had
| set up a proper republic. It had a constitution,
| elections, there were government posts etc. The problem
| was that the Party came first. The official that held the
| government post might've had the legal power, but in
| reality they were rubber stamps for the Party.
| heavyset_go wrote:
| Conviction rates are surprisingly high in a lot of the
| first world.
| Ma8ee wrote:
| I'm not sure what conclusion we should draw from that
| isolated fact. It can mean that people are convicted on
| too weak evidence, and it can mean that people are only
| put on trial when the evidence already is very strong.
| simple_phrases wrote:
| I think we can look at the plea bargain rates and assume
| that most people do not have the resources to stand
| trial, and thus plea out whether or not the charges
| against them are valid.
| bardworx wrote:
| Ok, maybe I'm naive, but why is that obvious?
|
| Not arguing just trying to understand.
|
| My rationale is: prosecutors have to weigh an "easy"
| conviction vs the possibility that labs that always
| guarantee results may be biased. In the event that they
| get caught up with a biased lab, wouldn't it look bad for
| their reputation? As in, they can't even perform due
| diligence on their sources, how can you trust them in a
| promoted role?
|
| Or am I just naive on how the legal system works?
| the_local_host wrote:
| I think you're underestimating the usefulness of
| scapegoats. If biased results exist, a lot of people look
| more effective. If such a bias is discovered, only a
| subset of those people will be blamed.
| eptcyka wrote:
| Prosecutors don't seek justice, they seek a conviction.
| benlivengood wrote:
| > In the event that they get caught up with a biased lab,
| wouldn't it look bad for their reputation? As in, they
| can't even perform due diligence on their sources, how
| can you trust them in a promoted role?
|
| I am not a lawyer but as I understand the U.S. criminal
| justice system (which is adversarial between defense and
| prosecution with the judge officiating and the jury
| picking the winner) prosecutors are not experts in
| particular technical fields which is why courts allow
| expert witnesses. The defense is responsible for calling
| their own experts who can dispute other biased witnesses.
| If no one successfully disputes an expert witness over
| time the prosecutor's trust in them is very likely to
| grow.
|
| A prosecutor will always claim they did due diligence by
| selecting a reputable expert witness up until it's proven
| that the witness was not in fact reliable, but claim
| (likely in good faith) that they had no way of knowing
| the expert witness was biased.
|
| A better solution might be to mandate that expert
| witnesses get independently tested but again that's
| something a defense council could facilitate and bring
| into evidence. If no defense team finds it valuable to do
| this kind of work then it's unlikely a prosecutor will
| see it as useful either. Lack of challenges implies that
| the evidence is more or less indisputable from a legal
| perspective.
| nobody9999 wrote:
| >I am not a lawyer but as I understand the U.S. criminal
| justice system (which is adversarial between defense and
| prosecution with the judge officiating and the jury
| picking the winner)
|
| That process happens very rarely. Federal criminal cases
| are resolved via plea bargain in ~97% of cases and state
| criminal cases are resolved via plea bargain in ~94% of
| cases[0].
|
| This is a travesty of justice, especially since most
| defendants are severely over-charged and often face long
| prison sentences if they actually insist on (and can
| afford) a trial.
|
| Those practices, along with cash bail, force even
| _innocent_ people to plead guilty to avoid having their
| lives destroyed by bankruptcy, loss of employment, homes
| and even custody of their children.
|
| And once they have a criminal conviction, they are
| stigmatized for life and are shut out of many jobs.
|
| In most of these cases, the forensic evidence (if any) is
| never presented. For a discussion of this, as well as
| American forensic practices, see _Blood, Powder and
| Residue_ [1], by Beth Bechky (Author discussion can be
| found here[2]).
|
| While there are no _required_ standards for forensics
| labs in the US (with some exceptions[4]), there are
| accreditation programs (example[3]), and not all forensic
| laboratories are for-profit companies.
|
| I'm not defending the quality or independence of any
| particular forensic lab, but it's definitely more diverse
| than just a bunch of corrupt, rapacious scum sucking at
| the teat of prosecutors' budgets.
|
| That said, most criminal defendants are at a significant
| disadvantage when it comes to performing/challening
| forensic research, as many state and federal labs provide
| such services for prosecutors, while defendants need to
| pay, often through the nose for them.
|
| It's just another way the US "justice" system is stacked
| against criminal defendants.
|
| [0] https://www.nbcnews.com/think/opinion/prisons-are-
| packed-bec...
|
| [1] https://press.princeton.edu/books/hardcover/978069118
| 3589/bl...
|
| [2] https://www.c-span.org/video/?508130-1/blood-powder-
| residue
|
| [3] https://anab.ansi.org/en/forensic-accreditation
|
| [4] https://www.justice.gov/opa/pr/justice-department-
| announces-...
|
| Edit: Added notation for reference [1]
| mulmen wrote:
| I wonder how much of this has to do with funding
| differences. Public defenders are famously underfunded
| but prosecutors have essentially unlimited resources. As
| a result they are on a fast-track to political fortune,
| especially if they "do their job well" and "win" (get a
| lot of convictions).
|
| How can the truth-finding benefits of the adversarial
| system be preserved while leveling the playing field by
| equalizing resources?
|
| My naive thought is that both parties pay into a common
| fund that is split equally between the prosecution and
| defense. But that seems like it has it's own set of
| pitfalls. Are there other models we can look to for
| ideas?
| horsawlarway wrote:
| I'll take a different approach than the other folks
| responding to you -
|
| There are plenty of profit motives involved that have
| little to do with securing future contracts with police
| departments.
|
| At the most basic - Source code reviews are expensive.
| They can often throw thousands of false flags that
| developers have to individually track down and verify are
| not problems.
|
| Then there's the reputation hit of admitting a mistake -
| If you've had a bug for the last 5 years that makes you
| less reliable, that alone _is_ likely to impact future
| contracts. So given the option of revealing this,
| admitting the mistake, and tackling the cases it
| impacted... OR... simply sweeping it under the rug and
| fixing it internally - some companies will take option
| number 2 (possibly most companies).
| geocar wrote:
| If you knew someone was guilty, because of a
| preponderance of other evidence you have the utmost faith
| in, such as a professional law enforcement officers'
| eyewitness account, and the DNA testing vendor you were
| evaluating said "not guilty", what exactly would you
| expect to happen?
| boomboomsubban wrote:
| That the eyewitness account would be discarded, they make
| for terrible evidence anyway.
| tekromancr wrote:
| I think it is REALLY hard to overstate how dangerously
| naive a view of the american justice system that is
| boomboomsubban wrote:
| I don't understand what you mean. I would expect them to
| make the obviously correct decision, though I'm fully
| aware that they very often don't.
| tsimionescu wrote:
| It depends on the definition of 'expect'. You should
| expect it of them, as in consider it unethical if they
| don't. You shouldn't expect them to live up to this
| standard, as in, if you were to bet on it, you'd likely
| lose.
| pelliphant wrote:
| That seems plausible.
|
| I would argue that if prosecutors and/or cops get to pick
| the tool, false positives would result in more sales.
|
| Just as false negatives would result in more sales if the
| defendant gets to make the pick.
| thgaway17 wrote:
| _false positives would result in more sales._
|
| That seems highly unlikely. Prosecutors, like all
| attorneys, talk. It wouldn't take too many getting burned
| by a false positive before word would get out (regardless
| of any nda).
|
| For example, once an expert gets Dauberted, you've got to
| think long and hard about being the next guy to use them.
| pelliphant wrote:
| But how would one actually know if a result is a false
| positive or not?
|
| Sure, in extreme cases it might be obvious, but if you
| just know that one test gives more matches than another,
| not knowing which one is the one giving false results?
| [deleted]
| 6510 wrote:
| A product that doesn't work wouldn't be considered.
| merpnderp wrote:
| But genetic evidence is often enough to convict by
| itself. A false positive is far more likely to land a
| conviction than an embarrassment to the prosecutor. I
| mean we live in a world where DNA evidence will easily
| override solid alibis.
| ncallaway wrote:
| You're a prosecutor. Your goal is to get convictions. You
| are objectively graded largely on your ability to secure
| convictions.
|
| Every incentive you have leads you to preferring
| convictions.
|
| You have a choice between two labs to hire for your field
| test. Lab A and Lab B.
|
| Lab A gives you conclusive evidence leading to a
| conviction 95% of the time. Lab B gives you conclusive
| evidence leading to a conviction 65% of the time. Price
| and speed are roughly comparable between the labs. Which
| lab do you select?
|
| ---
|
| You are now a lab director. You are under pressure to
| improve the sales of the lab's forensic services. You
| have pretty much maxed out making changes to improve
| price and speed, and have been matched by other labs. You
| understand very well the decision making process
| prosecutors will use to hire your lab.
|
| ---
|
| The incentives clearly lead prosecutors to pursuing labs
| that deliver better results. Labs know this, and so are
| under market pressure to delivery convictions for
| prosecutors.
| buttercraft wrote:
| Paying investigators _per conviction_ is a sure way to
| create incentive.
| Eric_WVGG wrote:
| Exactly. "Product X caught 25% more bad guys than Product
| Y" this stuff just writes itself.
|
| Policing is driven by numbers. There was a whole season
| of The Wire about this.
| ashtonkem wrote:
| Yes, yes they do. Prosecutors often _campaign_ on being
| "tough on crime", and even judges (where elected) are
| known to sentence more harshly during election season.
|
| The ability for an expert witness to deliver more
| convictions, even if by lying, is a feature for
| prosecutors, not a bug.
| AntiImperialist wrote:
| Two reasons:
|
| * they come across as effective if they "solve" the case
| i.e. they find someone guilty
|
| * it's practically impossible to challenge their findings
| because they hide their bs behind "experts" and "science"
| and other terms used by bs vendors
| irjustin wrote:
| A lot of DA and other prosecutors are measured by the
| number of convictions they get vs the number of cases they
| take to court.
|
| Their goals are always to get plea bargain because that is
| automatic guilt or if they take it to court, they'd better
| win. That's a lot of state resources just to lose.
|
| Overtime, you'll get people willing to bend the rules, turn
| a blind eye to 'weak evidence' or whatever.
|
| You're only as good as your measurement/incentivisation.
| treeman79 wrote:
| Deal: Parole for a year. If not, full resources of the
| government to convict you, jail time in the decades. Evan
| a failure to convict can ruin you for life.
|
| Even innocent people with a good alibi are going to take
| the deal.
| dimitrios1 wrote:
| This is a long tail regulatory force, but unfortunately,
| does nothing to stop bad actors from jumping in and making
| a quick profit for as long as they can until it goes
| noticed.
|
| Society also no longer has tolerance for processes that are
| not fully auditable and transparent, especially when it
| comes to our already marred justice system.
| avs733 wrote:
| The risk to their business of negatives whether or not they
| are false negatives, is likely higher. The resource
| asymmetry between a tax payer funded prosecution and a
| defendant funded defense is enormous.
| JoeAltmaier wrote:
| While there have been cases of technicians guilty of
| systemic fudging of data, that is a tiny exception to the
| vast majority of honest technicians with self-respect.
| kingkawn wrote:
| Your sentimentality is not enough for a good justice
| system.
| JoeAltmaier wrote:
| On the contrary, that's what the American justice system
| is entirely based upon. Fair judges, juries of peers and
| so on. Its the bedrock of justice.
|
| Engineers have some notion that it can all be replaced
| with science. In so far as science can certify the
| reliability of tests, that is good. But in the end one
| has to trust the humans between those tests and the
| courtroom.
| tsimionescu wrote:
| The whole point of this discussion is that this trust is
| being largely misplaced.
|
| A lot of forensic science in itself is essentially
| phrenology (tooth prints, hair analysis - the non DNA
| kind, even fingerprinting is of exaggerated value). A lot
| of labs systematically turn out biased results with
| generally no expectation of risk, either personally to
| researchers and definitely not to higher ups.
|
| Ultimately, forensic evidence should be seen as a signal,
| but not nearly as trustworthy as witness testimony (which
| in itself is not very trustworthy), despite what many
| believe.
|
| And this is important for the general public to
| understand, the people who will participate in juries.
| The mystique that forensic experts have can make juries
| give extremely wrong decisions (even bad aquitalls on
| lack of forensic evidence).
| JoeAltmaier wrote:
| Yes there is a need for controls and independent
| certification.
| tsimionescu wrote:
| There is also a need to recognize that a lot of this
| science just doesn't work (to the extent that it is often
| presented). The right way to phrase a lot of the
| conclusions of these experts, at the state of the art,
| would be 'in my expert opinion, it _looks like it might
| be_ their DNA /fingerprint/hair/teeth', not 'in my expert
| opinion it is [...]'. Because even the state of the art
| is often about at that level - probably closer to 75%
| accuracy than 99.99% like it's often treated. Especially
| on real world, partial, corrupted samples.
|
| And to emphasize again, I'm talking about the state of
| the art without the biases being discussed. The biases
| only compound this problem significantly.
| macinjosh wrote:
| Prosecutors want a 100% conviction rate. They are the ones
| who hire these companies. Generally speaking companies give
| customers what they are looking for.
| chopin wrote:
| Why should that come to light if the process is opaque?
| Most defendants don't have the resources to challenge that.
|
| And if it comes to light, acknowledge a one-time error and
| carry on.
| wtetzner wrote:
| I seems like there are no repercussions for sending an
| innocent person to prison. I don't know if that's _true_ ,
| but it definitely _seems_ like it.
| jimz wrote:
| Depends on what you mean by repercussions. Since there is
| at least informal and sometimes fairly open scorekeeping
| happening at prosecutor's offices there's some hurt pride
| if your conviction is overturned later. Everything else
| either gets taken care of by the taxpayer or prosecutors'
| immunity for their official acts so on a personal level,
| no, there won't be any actual repercussions.
| linuxftw wrote:
| Perhaps in a just society. Most places, the state and it's
| agents can do no wrong, there's almost no incentive to do
| the right thing.
| andi999 wrote:
| The prosecution might call in a different expert in the
| next trial if the current one is not 'good'.
| refurb wrote:
| Then they get hired by the defense.
| MaxBarraclough wrote:
| The defence can rarely afford their services.
| jimz wrote:
| Prosecutors aren't necessarily blessed with bigger
| budgets in a lot of places, they simply have a
| separately-funded (and very well funded at that)
| investigative body that is the police or sheriff's office
| doing the really expensive part that the defense,
| especially a public defender's office, will only have a
| skeleton crew to do. The problem is that defense may not
| have the resources or knowledge to use inadequacies of
| something like this as a defense in the first place, and
| may not be able to effectively cross or direct an expert
| because of the lack of specific expertise and the
| reactive nature of the job.
|
| It's not that the venn diagram of tech-literate and
| criminal-defense lawyers are entirely separate circles,
| but having been "that guy" in a public defender's office
| for even basic stuff like cell tower triangulation
| accuracy to finding proprietary surveillance video codecs
| to decode exculpatory evidence, you really need to start
| at square one while the prosecution have the whole police
| department's resources, expertise, and initiative at
| their disposal. You can afford the expert, it just won't
| do any good when you don't know what questions to ask
| that will actually be effective. And unsurprisingly those
| who do have experience in technical, specialized fields
| tend to get poached into the private sector or out of
| trial (really plea) practice all together, so the
| knowledge/bullshit gap will still exist and there's no
| real consistent way to bridge it.
| andi999 wrote:
| Also the defense is many customers while the prosecution
| is bulk.
| heavyset_go wrote:
| They fill a similar role to prosecutors that field drug tests
| serve to cops[1]: they aren't meant to be accurate, they
| exist to give the cops legitimacy as they proceed to do
| whatever it is they wanted to do to you. In the cops' case,
| they need legitimacy to search and arrest you, and in the
| prosecutors' case, they need legitimacy to charge and convict
| you.
|
| If field drug tests were actually accurate, they wouldn't be
| bought and used. Similarly, if an expert witness fails to
| give prosecutors the results and testimony that they want,
| they wouldn't be hired again.
|
| [1] " _How a $2 Roadside Drug Test Sends Innocent People to
| Jail_ ": https://www.nytimes.com/2016/07/10/magazine/how-a-2-
| roadside...
| jdsalaro wrote:
| > They fill a similar role to prosecutors that field drug
| tests serve to cops[1]: they aren't meant to be accurate,
| they exist to give the the cops legitimacy as they proceed
| to do whatever it is they wanted to do to you. In the cops'
| case, they need legitimacy to search and arrest you, and in
| the prosecutors' case, they need legitimacy to charge and
| convict you.
|
| In case any of you are interested, or completely
| flabbergasted as I was, by the idea that law enforcement's
| purpose and raison d'etre could ever become as distorted
| and contorted as this comment mentions, you should
| definitely read more on Walter "Johnny D." Macmillan [1] or
| watch the movie based on his story: Just Mercy[2].
|
| Absolutely mind-blowing stuff.
|
| [1] https://en.wikipedia.org/wiki/Just_Mercy
|
| [2] https://en.wikipedia.org/wiki/Walter_McMillian
| bluntfang wrote:
| Wow, think about what it takes to be an elected official
| (Sheriff) and not only perjure, lie, and coerce
| witnesses, but be outed for all of those crimes, and then
| continue to run for election and continue your tenure for
| decades. How does someone live with themselves knowing
| they purposely put an innocent man on death row, and then
| CONTINUE YOUR CAREER, probably thinking yourself as
| successful?
|
| The guilt I feel when I introduce a software bug that
| effects my corporate customers can sometimes burn my
| motivation for WEEKS. Who are these people?!
| neodymiumphish wrote:
| This absolutely depends on the agency/situation.
|
| My agency used to rate our agents based on the amount of
| cases that led to convictions. Years back, we recognized that
| disproving an allegation was equally worthwhile, and settings
| promotions/evaluations based on "proving or disproving" the
| allegation was much more objectively reasonable than focusing
| strictly on convictions. There is still a reference to case
| completion speed, value of recovered money/property, and
| conviction results (providing a thorough enough investigation
| that the suspect is convicted for X years, for example), but
| disproving an allegation or proving that a different person
| committed the crime is far more appreciable to the agency.
|
| We're fortunate in that we are a federal agency, instead of
| local/state level, where they can be significantly more busy
| with countless lesser offenses, while still running some
| extremely high-profile stuff, but I think it would be a huge
| boon to law enforcement if states mandated an approach
| focused on this objective metric that isn't strictly on
| "getting him".
| ChrisRR wrote:
| This is similar to a case a few years back where someone asked to
| see the source code of a breathalyser that had found them guilty.
|
| AFAIR, the breathalyser was incorrectly averaging the readings,
| giving disproportional weight to the first reading.
|
| I don't know if it was enough to rule in their favour, but I'm
| sure it called the data into question
|
| Edit: Looks like it was a Draeger breathalyser
| https://www.schneier.com/blog/archives/2009/05/software_prob...
| text70 wrote:
| A tool designed to find people guilty is biased to find people
| guilty.
|
| As far as I know it is fairly easy to take a generic dna
| sequencer meant for healtcare diagnostics, and repurpose it for
| STR analysis. The only major difference between the healthcare
| versions and the forensic versions is the software i/o.
| sdflhasjd wrote:
| > A tool designed to find people guilty is biased to find
| people guilty.
|
| I don't see those particular issues make it biased, just
| inaccurate - it could go either way.
| radu_floricica wrote:
| And yet somehow whenever you take a closer look at
| mislabeled product prices, the average is always in favor
| of the store. And that's far from the only industry.
|
| Complex tools are the product of many thousands of
| individual decisions taken by humans, humans aware of who's
| the paying client.
| jstanley wrote:
| > And yet somehow whenever you take a closer look at
| mislabeled product prices, the average is always in favor
| of the store.
|
| This could just as easily be selection bias: the errors
| in favour of the customer are less likely to get reported
| by customers.
| sdflhasjd wrote:
| I still believe Hanlons razor applies. I've seen products
| that have serious performance affecting bugs caused by
| similar mistakes.
| ncallaway wrote:
| I still think--even when applying Hanlon's razor--there's
| an imbalance in incentives that leads to a weight in
| favor of the interests of the party paying for the test.
|
| Take the store pricing example. Suppose the store's
| pricing & labeling process produce an equal number of
| bugs at checkout in favor of the store and in opposition
| to the store.
|
| The store is heavily incentivized to _detect_ the errors
| that are opposed to them. They are much less likely to
| detect the errors in their favor. Consider the manager
| that looks at the cash at the end of the day and notices
| they are $500 short. They likely dig hard to find the
| root cause of the issue, detect the pricing disparity and
| correct it. Now consider the manager that is $500 over at
| the end of the day. They are much more likely to say:
| "that's weird", shrug their shoulders and move on.
|
| The same applies to forensic tools. Even if they
| originally produced bugs in both directions, their own
| internal QA and the market of police officers are likely
| to work hard to detect bugs that make them less likely to
| allow them to make an arrest.
|
| The net result is that the tools end up with a bias in
| one direction, even if the original developers made an
| equal number of mistakes in both directions.
| boyesm wrote:
| > And yet somehow whenever you take a closer look at
| mislabeled product prices, the average is always in favor
| of the store.
|
| What is this based on?
| sdenton4 wrote:
| I had a 'fun' experience along these lines with health
| insurance and medical bills a couple years ago. I can
| confirm that in our case at least, /every/ error we found
| was not in our favor, and took usually about an hour on
| the phone to get fixed.
|
| The somewhat-less-malicious interpretation is that the
| companies have a strong incentive to detect + fix errors
| that cost them money. Meanwhile, consumers are a) non-
| centralized, uncoordinated, and often unaware of errors,
| and b) have no way to fix systemic issues that impact
| them. And the companies therefore have no /real/
| incentive to fix systemic problems. It is literally more
| profitable to fix the bills of the few people who
| complain, as they still make money on the remainder who
| don't notice the errors in the first place.
|
| (on edit; exactly what the other comment one subthread
| over said. :P )
| bgirard wrote:
| When running an experiment and following poor practices
| (i.e. p-hacking), results that fit the hypothesis will be
| accepted more readily and negative results will be debugged
| or re-ran more often.
|
| i.e. The initial error may be randomly distributed. But the
| follow-up on the error will have a lot of bias.
| throw2838 wrote:
| Draeger requires calibration prior each use for temperature and
| humidity. It is easy to get it thrown out as evidence.
| lwigo wrote:
| When the officer shows up and swears under oath that he
| "calibrated it that morning" it's not so easy. :shrug:
| jdironman wrote:
| Temps vary throughout the day.
| lwigo wrote:
| A rural small town judge probably doesn't have the best
| insight into that, just that Corporal Bubba said he
| calibrated it so it must've been right.
|
| I know, this is exceedingly cynical.
| klyrs wrote:
| Your "Corporal Bubba" isn't just cynical, it leans
| heavily upon a harmful stereotype that folks in small
| towns are uneducated simpletons. Stop the polarization,
| please
| zymhan wrote:
| I'd love it if more cops had college degrees in the US,
| but they don't. Ignoring the problem doesn't make it go
| away.
| owl57 wrote:
| Why wouldn't they be? I went to university with people
| from all over Russia. They don't tend to return to their
| home towns after getting educated. Does this work
| differently in America?
| zymhan wrote:
| > Does this work differently in America?
|
| No, not really.
| [deleted]
| hluska wrote:
| Sadly, when you consider the available evidence, Corporal
| Bubba isn't too far from the truth. I say that as someone
| whose Dad is a retired police officer and who spent most
| of his youth in a very small town with sub-1200 people.
| OnACoffeeBreak wrote:
| Logs of calibration procedures along with calibration
| results should be stored on the device and auditable along
| with test results.
| dathinab wrote:
| I wouldn't be surprised if "incorrectly averaging" and similar
| are very common software errors.
|
| The reasons are manifold, including:
|
| - Normalized values need to be averaged differently the
| absolute values.
|
| - Floating point has limited precision, even just correctly
| summing/multiplying numbers need special care if you care about
| correctness. Results can, in the worst case, be of by a massive
| amount.
|
| Often you don't need to care about it so it's not uncommon for
| especially junior programmers to be not so aware about it.
|
| I mean in the last 3 years of working as a professional
| software engineer/developer I didn't need any of this at all,
| but once I do I know what to look out for.
| qwertox wrote:
| Here in Germany we have somewhat similar cases, but where the
| accusation is way less damaging than the case of this article, in
| which a false positive would have the drastic result of being
| labeled a murderer.
|
| The cases are related to new speeding cameras which work with
| laser, where the defendants are complaining that these new
| devices are black boxes, and that they demand access to the raw
| data which these devices process. The problem is that these
| devices discard the raw data after having processed it and come
| to a conclusion that the driver was or was not speeding.
|
| The devices in question are Traffistar S350 from Jenoptik and
| PoliScan SM1 from Vitronic.
|
| There were discussions about a required software update which
| retains all this data, but apparently the devices lack the
| storage capability to do so. The National Metrology Institute of
| Germany (Physikalisch-Technische Bundesanstalt (PTB)) responded
| to this, that they would not re-certify these devices with
| updated software because from their point of view they work "as
| specified".
| nolok wrote:
| Now all you need is the proper amount of collusion/corruption
| between the certifying agency and the manufacturer to have a
| magic box that does whatever the one paying the bills want.
| Might seem far fetched in a developped country, until you read
| about the Boeing/FAA thing happened.
| Kinrany wrote:
| Surely there aren't so many speeders that storing a few seconds
| per violation would add up to a significant amount.
| fxleach wrote:
| > it would take eight and a half years to review at a rate of ten
| lines an hour
|
| Wow, the co-founder's argument to not disclose the source code to
| the defendant was that it's too many lines of code. Also... ten
| lines reviewed in an hour!?
| meowster wrote:
| In the TV shows and books, opposing counsel always want to
| inundate the protagonist with truckloads of boxes full of
| printed papers when they could just handover a flashdrive
| instead. Are there any lawyers here who can speak to the
| accuracy of such portrayal?
|
| It would seem the next logical step would be for every other
| lawyer to say they shouldn't hand over discovery because it
| would take to long for the otherside when they bury them in
| paperwork.
| _Wintermute wrote:
| Not sure if it's the best source, but according to a "Youtube
| laywer reacts" video [0], it's a common tactic but you can
| complain about it to the judge and request documents in a
| more appropriate form.
|
| [0] https://youtu.be/spr5smxuO5E?t=1284
| marcodiego wrote:
| Wouldn't this set a precedent to eventually force any forensic
| software to become open source?
| jMyles wrote:
| It's unfathomable that the state can introduce evidence of any
| variety that _isn 't_ open source.
|
| Secret policing and secrecy in prosecution have no place in
| decent society.
| gsich wrote:
| Hopefully.
| vincnetas wrote:
| Code inspections is not same as open source.
| jimnotgym wrote:
| I guess open-source implementations of various algorithms that
| had been demonstrated effective, if produced by a not-for-
| profit maybe, could be very disruptive to this 'industry'.
| Zenst wrote:
| Not per-say, though it certainly affords it the same level of
| scrutiny.
|
| Which may well be a win as open source already has that.
|
| However, even open source has bugs that pass scrutiny as many
| CVE's can attest, so whilst a code review may find nothing
| wrong, that in itself could be used by a lawyer to create
| reasonable doubt, if the lawyer is good.
| [deleted]
| jlgaddis wrote:
| > _per-say_
|
| https://en.wiktionary.org/wiki/per_se
| schmorptron wrote:
| That'd be ideal
| chadlavi wrote:
| While this may lead to more transparent technology use in the
| justice system, a more realistic outcome is that everyone accused
| of any crime where a technology was used in the process of
| determining guilt (which could get quite picayune; did the
| prosecutor's office use Excel? Is that something the defendant
| could demand to check the source code on?) will use this as a way
| to slow-roll the process indefinitely.
| bagacrap wrote:
| Definitely if an excel macro determined my guilt based on a
| dataset, I would like to be able to read that macro.
| abfan1127 wrote:
| or at least double check the math.
| Spivak wrote:
| Yes? I mean the intimate details of how cell networks work have
| been the difference between guilty and not before when the only
| evidence is some black-box report generated by AT&T that seems
| more accurate than is actually guaranteed.
|
| This wouldn't be necessary if the expert on the stand was a
| geneticist who ran the test. But when the expert literally is
| the software you can't really cross-examine the company's sales
| rep.
| tehwebguy wrote:
| Well sure, if they rely entirely on Excel to generate and spit
| out an expert witness testimony. If that ever happens it will
| be extremely important to understand what's happening under the
| hood in Excel.
| Jolter wrote:
| Since when do defendants have an interest in prolonged
| sentencing processes? For major crimes like this, they are in
| jail with restrictions. They would get better treatment once
| sentenced, in the prison system, wouldn't they?
| [deleted]
| spaetzleesser wrote:
| There is so much junk science going on in forensics that it would
| be great to require everything to be open sourced. Same for
| voting machines and anything police in general is using
| (predictive policing is pretty scary). There is way too much
| stuff hidden and can be challenged only if you have very deep
| pockets.
| ineedasername wrote:
| If ever there was something that should be fully transparent it
| is the mechanisms by which a person might be found guilty of a
| crime. The defendant shouldn't even have had to fight for this.
| It should be a fundamental cornerstone of criminal prosecutions.
| meowface wrote:
| These cases also often come up with drug and alcohol detection
| tests, and as John Oliver points out in
| https://www.youtube.com/watch?v=1f2iawp0y5Y, software used to
| select jurors.
|
| All of these companies claim that their source code is valuable
| intellectual property and that disclosing it can hurt their
| business. Even if this were true, when you're providing
| something that can be a significant factor in someone being
| imprisoned or executed, when creating the business you should
| accept that you're providing a public service that needs to be
| publicly accountable.
|
| If it's not open source, at the very least there should be a
| requirement that software code and hardware designs must be
| provided on-demand to experts in court cases (with a non-
| disclosure clause to mitigate leaks and corporate espionage
| etc.).
| bargl wrote:
| Without jumping on the conspiracy bandwagon, I'd also like to
| see this applied to voting software. I know it's a hot topic,
| and I'm honestly not trying to get political.
|
| Software that is critical to our fundamental human rights,
| and is being used by our government should be open source, or
| at least audited by a group of people who sign Non-
| competes/NDA and can't go work for competitors, or with some
| other mechanism to protect IP that I can't think of.
| jedberg wrote:
| The beauty of voting software is that you don't have to
| verify the code if you hold the vote correctly. If the
| software provides a voter verifiable paper trail, the voter
| can verify their vote before turning it in.
|
| The county can then verify the software by manually
| counting a random selection of paper votes to see if they
| match the software. If they do, then the software is
| correct, otherwise it is not. You then have a full by-hand
| recount and tell the vendor to fix their software.
| jjeaff wrote:
| I agree, for my own piece of mind. But I am also certain
| that it would have made no difference in our current
| predicament with a third of the country thinking the
| election was stolen.
|
| It has been shown to us time and time again that no actual
| evidence is required to get people to believe what they
| want to believe.
|
| And the more technical the evidence (i.e. source code), the
| less helpful.
| bargl wrote:
| >But I am also certain that it would have made no
| difference in our current predicament with a third of the
| country thinking the election was stolen.
|
| It would have changed some peoples minds I don't know if
| the change would have been a few thousand or 10s of
| millions. I can't say if it would have a dent in the 1/3
| of people or not. I can't predict that. It would have
| helped me with my own peace of mind. And frank I think
| it's overall the right thing for us to do.
|
| >And the more technical the evidence (i.e. source code),
| the less helpful.
|
| Disinformation is powerful, I'm not suggesting this alone
| would fix that. I disagree that more technical evidence
| is harmful. Global warming is benefiting from
| transparency and evidence. It takes generations to change
| political will not years. The evidence there has shifted
| our whole economy, just maybe not fast enough.
|
| There will always, always be deniers. Global warming,
| flat earth, vaccinations, etc. Evidence _helps_ battle
| deniers in these areas, but it takes generations for
| these ideas to become mainstream and the deniers to go
| from 99% of people to 2% of people.
|
| Also, 2% of people think the earth is flat? Holy crap.
| https://www.sciencealert.com/one-third-millennials-
| believe-f...
| acct776 wrote:
| It shouldn't be considered conspiracy theory that,
| technically speaking, many things in our nation are an
| insecure joke, including our system of voting.
| alistairSH wrote:
| _...disclosing it can hurt their business..._
|
| And there's an entire body of law based around IP which they
| can use to protect their business, just like everybody else.
| zymhan wrote:
| Their business interests are also of minuscule importance
| compared to the impact to society these tools have.
| frongpik wrote:
| Nothing is going to change until this software convicts a 10M
| net worth dude for something he didn't do.
| rland wrote:
| These companies are disgusting. They peddle black box "models,"
| that essentially ride the good reputation of DNA as infallible
| (which it is most certainly not) to get convictions on dubious or
| no evidence.
|
| The way it works is that if there is a sample from a crime scene,
| they send it to these guys and they analyze it with their
| software to detect "statistical" DNA from the sample. These
| samples are the ones that are too crappy to actually make a
| definitive match -- they are a statistical match. So you say "I
| think Jim, Bob, and Alice were on scene," and it says "10%
| likelihood Jim DNA, 5% likelihood Bob DNA, 45% Alice DNA." Do you
| think it ever says "99% no DNA" in the sample?
|
| It's basically Theranos, except instead of wasting $50 on a
| shitty blood test you get life in prison.
|
| Ostensibly, it searches the entire DNA database for matches, and
| only returns a positive result if there's a positive match.
|
| But it's a statistical model, using inputs that are crappy at
| best (because if it was an actual DNA match, they would send it
| off to in house forensics who would be able to do PCR...) and
| which includes inputs from circumstantial evidence as priors.
| Like we believe Alice was at the scene therefore if you find any
| statistical likelihood that this is Alice's DNA boost that.
|
| They often run the model multiple times in a row, and use the
| result that the DA likes the most to enter into evidence. This is
| because the models return different results each time -- of
| course they'd say, iTs StAtiStIcaL, so they can do that...
|
| And the source code is completely impenetrable. They argue that
| it's a "trade secret" that jeopardizes their ability to make
| future profits, so it cannot be open-sourced. These guys could
| have a model that just says "what percentage should the thing
| read, Senor D.A.?" The entire product is a sham. And because it's
| 170k LOC, no one has the time or the qualifications
| (Judges/Attorneys reading source code? Yeah right!) to review it,
| even if it were open source.
|
| Pure quackery, and often times, decades-long sentences or life in
| prison for the defendant. These companies are pure filth worthy
| of the lowest revulsion. It's a wonder any convictions happen at
| all because of this stuff, but jurors have very inaccurate
| conceptions of forensic science, thanks to shit like CSI, Law and
| Order, etc. These companies happily play into that image and
| people really believe this stuff works.
| ABeeSea wrote:
| > Mark Perlin, is said to have argued against source code
| analysis by claiming that the program, consisting of 170,000
| lines of MATLAB code, is so dense it would take eight and a half
| years to review at a rate of ten lines an hour.
|
| So it's definitely riddled with bugs. And I can't imagine that
| much matlab code following rigorous software engineering
| practices.
| bryanrasmussen wrote:
| and surprisingly all of the code is of equal importance so you
| really need to review each line sequentially! Instead of
| finding stuff that you think is most likely to relate to what
| you're trying to figure out and debug from there. Wow I would
| like to see this marvel of engineering myself!
| Scandiravian wrote:
| Kind of makes me wonder: if the argument is that the code is
| too complex to review and understand, does that mean the
| company is not doing code reviews themselves?
| watwut wrote:
| You can do code review on every single commit while not
| being able to make overall analysis of the massive
| codebase.
| Scandiravian wrote:
| I completely agree, but if the reviewer isn't able to
| (with some amount of accuracy) predict the impact the
| committed code will have on overall behaviour, then
| there's very limited value on doing the review in the
| first place
| watwut wrote:
| In any larger project, the reviewer are not able to
| predict impact from reading commit.
|
| More importantly, typical reviewer have only small partia
| area where he has good idea about which commit is bad
| idea. He however does not understand whole codebase.
|
| Knowing what the whole does and knowing what my module
| does are two different things.
| Scandiravian wrote:
| I again completely agree with you
|
| Looking back at my reply, I think I should have added a
| bit of background to clarify my comment
|
| My master's degree is in bioinformatics and I worked in
| the biotech industry until about a year ago. I mainly
| worked as a consultant for top 20 pharma companies, but
| also did work on different in-house projects and in
| academia
|
| From my experience in the industry, I find it very
| unlikely that the software mentioned in the article is
| structured in a modular way. I've yet to see good
| software practices outside one or two academic projects.
| Most pharma companies still use copying and renaming
| folders as version control. Naturally I'm sceptical of
| any code coming from the biotech industry
|
| On top of that, it's written in MATLAB. I have only ever
| seen this used by statisticians and university
| researchers, never by software engineers
|
| I'm therefore willing to bet, that when the reviewers
| open the source code, they'll find unstructured mess of
| spaghetti code, that has never been refactored, reviewed
| or tested
|
| So yes - I agree in all your points, but I find it
| unlikely that they're being applied to this particular
| project
| bryanrasmussen wrote:
| this has actually only happened to me a couple times but it
| has happened - someone tells me Bryan, go look at the code
| X did in Y, figure out if we refactor. X would then tell me
| - that code is really complicated is full of algorithms! I
| go and look at the code realize that for what it is trying
| to do can be cut down from 10 pages of printed code to less
| than 1 and it was incredibly simple what actually needed to
| be done.
|
| In short when someone tells me the stuff is too complicated
| because too clever and advanced I tend to disbelieve them.
|
| that said I have of course written my too complicated stuff
| lots of times, but if asked I don't say it was because I'm
| clever.
|
| names anonymized so as to not accidentally hurt anyone's
| feelings.
|
| on edit: actually one time the code was clever but not
| especially difficult, they just used the algorithms line
| because they didn't want anyone messing with their stuff.
| Scandiravian wrote:
| I think there's a bias towards judging things to be
| "clever" if they're hard to understand
|
| It's a cliche to have a "what idiot wrote this" outburst,
| then realise it's your own code, because most of us have
| written our fair share of "clever" code
|
| My boss explicitly stated that he doesn't want to see any
| "clever" or "smart" code in our product - write code
| based on simple fundamentals, benchmark before deciding
| to optimise, and be respectful in your reviews
|
| I like my boss a lot
| michaelmior wrote:
| If the code was developed in less than 8.5 years, it seems
| like the answer must be yes based on their previous claims.
| gambiting wrote:
| Well I mean if something requires 8 man years of work you
| can do it in a year with 8 people. Not always, but
| usually.
| markwkw wrote:
| I would claim that if something requires 8 man years,
| that it will _most definitely_ take more than a year to
| develop with 8 people.
|
| Communication takes time, coordination takes time, there
| is an incremental cost to each news person added to a
| team. From experience, perhaps with 2-3 people who happen
| to gel well together you may get close to proportional
| scaling of output, but with 8 it's really unlikely in the
| real world.
| watwut wrote:
| I would find it extremely unlikely to work that way.
| agravier wrote:
| On the contrary, such linear scaling would be quite
| exceptional. I'm speaking from experience but you don't
| need to trust me; I invite you read any book on software
| engineering management, starting from The Mythical Man-
| Month by Brooks.
| bryanrasmussen wrote:
| the mythical man month was first published in 1975, I
| think the typical applications programmers work on today
| have changed significantly since then and encompass many
| different disciplines (to be thought professional) - so
| many disciplines that one developer is likely to be the
| master of all. It is true that there is a communication
| overhead to adding more people so it will not scale
| linearly, but if a single developer has taken 8 years to
| build something in our era it seems likely that having 8
| people might get it done say 1 and a half to two years.
| syshum wrote:
| I also cant image anyone thinking that would be a winning legal
| argument... "This software is too complex to look at so just
| trust us" Really... that is what they went with...
| avs733 wrote:
| unfortunately, the technical illiteracy of much of the US
| judiciary has made this effective.
| delecti wrote:
| Yeah, that's one of those "defenses" that feels like a
| confession.
| Scandiravian wrote:
| My exact thoughts. This sounds like a classic example of
| launching a prototype created by domain specialists
| (biostatisticians and bioinformaticians) as production software
| and skipping on the expensive stuff, like sound development
| practices
| antattack wrote:
| For reference, whole Tesla Autopilot is 'few hundred
| thousand' loc.
|
| https://youtu.be/YAtLTLiqNwg?t=947
| isolli wrote:
| It also sounds exactly like Ferguson's Imperial College
| epidemiology model that apparently compelled politicians into
| imposing hard lockdowns (and was likely wrong by at least an
| order of magnitude):
|
| - "a single 15k line C file that had been worked on for a
| decade" [0]
|
| - code review of the model: [1]
|
| - corresponding HN discussion: [2] (including sad appeals to
| authority: you're not an epidemiologist)
|
| - other HN discussion [3] (including ridiculously blaming
| programmers for making C++ available to non-programmers)
|
| [0]
| https://twitter.com/ID_AA_Carmack/status/1254872369556074496
|
| [1] https://lockdownsceptics.org/code-review-of-fergusons-
| model/
|
| [2] https://news.ycombinator.com/item?id=23093944
|
| [3] https://news.ycombinator.com/item?id=23222338
| gww wrote:
| There is so little emphasis in production software
| development in bioinformatics and biostatistics. Despite a
| lot of groups open sourcing their code it's is nearly
| unusable and not reproducible due to hard coding, ignoring
| edge cases, and dumping the majority of the code on a single
| giant R or python function.
| Scandiravian wrote:
| That's my experience as well. My master's is in
| bioinformatics and I worked for several years in biotech.
|
| I got frustrated because my concerns that my team's
| development practices were causing issues on a regular
| basis, were ignored. I was continuously able to predict
| what issues we would run into, but no-one seemed to care -
| I even had a manager tell me, that it was good that our
| software was buggy, since the client would continue paying
| us to fix it
|
| I've since left the biotech industry. There's a limit to
| how many times I want to run my head against that
| particular wall
| natechols wrote:
| It's a real problem, and I've been struggling with it for
| two decades, but even so I am legitimately impressed (and
| not in a good way) if they have 170,000 lines of Matlab
| code in their production software. That takes a really
| special combination of productivity and cluelessness, even
| for academic specialists. Regardless of the facts of this
| particular case, it should be absolutely horrifying that
| anyone's freedom is left up to a gigantic pile of unaudited
| Matlab code. (That said I am almost certain he added some
| zeros to the number, I have a hard time imagining what
| they're doing that could be that complex.)
| Scandiravian wrote:
| Choosing MATLAB as a language for software that could
| potentially lead to people dying (in areas where they
| still have the death penalty) is a gigantic red-flag
|
| I don't know how many job postings ask for a software
| engineer who knows MATLAB, but I can't recall any
| bitexploder wrote:
| I have done a lot of source code review in my time. For
| security assessments. Our general rule of thumb was about 10k
| lines per week that we can really get deep on. 10 lines an hour
| would only be for the most dense code and critical path stuff.
| They will need a reviewer that knows the domain (DNA), but it's
| perfectly reasonable to review that code on a weeks/months time
| scale, definitely not years.
| partyboat1586 wrote:
| >170,000 lines of MATLAB code
|
| This is a deep problem. Many scientists don't understand
| software engineering and more and more need to write bigger and
| bigger programs. And most of the time they don't open source
| their code.
|
| Open source science.
| moron4hire wrote:
| That's like an accountant, accused of embezalling, refusing to
| hand over the ledgers because "there are just too many records
| to go through". Yeah-no, that's kind of the whole point. We
| want to find out what you've been hiding in that wall of paper.
| phire wrote:
| _> The co-founder of the company, Mark Perlin, is said to have
| argued against source code analysis by claiming that the program,
| consisting of 170,000 lines of MATLAB code, is so dense it would
| take eight and a half years to review at a rate of ten lines an
| hour._
|
| First, the defence doesn't necessarily have to evaluate all
| 170,000 lines. They just need to find one buggy line which could
| potentially overturn the result.
|
| Second, even if it did take a full 8 years, is that a good reason
| to deny the defendant due process?
| buzzerbetrayed wrote:
| Wait until you hear how long it would take at a rate of 2 lines
| per hour, as long as we're just throwing out random numbers.
| Cthulhu_ wrote:
| > Second, even if it did take a full 8 years, is that a good
| reason to deny the defendant due process?
|
| No, but the person that wants to have it analyzed will have to
| either spend the time themselves, or pay the expert witness for
| their time; it could be a costly affair.
|
| But I think it's warranted. An independent software review, and
| a double blind assertion with the exact version of the software
| used in the conviction to test the accuracy and reliability of
| the application.
| edgyquant wrote:
| Any software used to convict people, especially on a such a
| serious crime, should be audited like the fed is. Twice
| yearly, once by a public firm and another by the government
| itself. It should have to pass both of these audits to be
| used.
| kspacewalk2 wrote:
| >Second, even if it did take a full 8 years, is that a good
| reason to deny the defendant due process?
|
| "It's just gonna take _so long_ , plus the code is a bit messy.
| We're gonna be doing all that _work_ just because the rest of
| someone 's life teeters on the results of the inquiry? Maan,
| that's a bummer."
| breck wrote:
| Lol. If it would take 8.5 yrs to review, it's probably god
| awful, and should never ever ever be used to convict someone of
| such a crime.
|
| My prediction: this firm will probably try to get removed from
| the case, rather than open source their shitty code.
|
| Source: I've worked on MATLAB codebases for various genomics
| research projects in the past.
| nickodell wrote:
| >My prediction: this firm will probably try to get removed
| from the case, rather than open source their shitty code.
|
| That isn't necessarily their choice. The prosecutors will
| make the decision about whether to withdraw the DNA evidence.
| They probably won't, given that they would need to give the
| defendant a new trial, which could lead to an accused
| murderer getting off. A bad look for any prosecutor.
|
| More to the point, if the firm withdraws from any case where
| their credibility is questioned, what does that say to law
| enforcement agencies who are thinking about using their
| software?
| ahepp wrote:
| My understanding is that (some) law enforcement agencies
| have been more than happy to drop cases rather than subject
| investigative tools to proper scrutiny[0]. They have no
| qualms resorting to "parallel construction"[1], and simply
| using the inadmissible (sometimes illegal) evidence to find
| admissible evidence.
|
| [0] https://arstechnica.com/tech-policy/2015/04/fbi-would-
| rather...
|
| [1] https://en.wikipedia.org/wiki/Parallel_construction
| gidan wrote:
| That would be implying that the prosecutor would prefer
| taking the life of an innocent rather than having it hurt
| his career, making the prosecutor kind of a criminal.
| melq wrote:
| >making the prosecutor kind of a criminal.
|
| Never met a lawyer before huh?
|
| Jokes aside, prosecutors pushing through cases they know
| to be unsound isn't exactly uncommon. Many prosecutors
| are more concerned with their conviction rates than they
| are in justice, because that's what they are measured and
| rewarded by.
| rtx wrote:
| I often hear this, who is rewarding them for high
| conviction rate.
| vkou wrote:
| Voters, because when it comes to issues of criminal
| justice, crowds are rarely paragons of sober temperance
| and restraint.
| jabits wrote:
| I think you are wrong and that most prosecutors want to
| do the right thing, like most working people
| hluska wrote:
| Sadly, evidence contradicts that thought. It shouldn't
| but it does.
| hutzlibu wrote:
| Everyone wants to do the right thing.
|
| It is just that some think the right thing for themself
| is to maximize their career progress.
|
| And I would not know in general about state prosecutors,
| but what I know anecdotally second hand, does not sound
| good.
| melq wrote:
| I believe that's true as well, and I never said
| otherwise.
| JakeTheAndroid wrote:
| "Right" and "wrong" are dependent upon the system and how
| it rewards you.I would agree that most prosecutors what
| to serve justice for malfeasance that has been committed.
| That's different than whether a case is the "right" or
| "wrong" one to take.
|
| If a case seems unclear, and you could spend years
| working on a conviction that will ultimately fall
| through, that hurts your ability to do justice for more
| readily winnable cases. You have to spend the time
| building a case, do all the paperwork, go to trial, etc.
| That's opportunity cost. So spending that on a case you
| have 10% chance of winning just isn't a good use of time.
| Add that to the fact that conviction rate is a metric
| used to quantify skill, you're rewarded for serving
| justice successfully. And that then dictates how much
| money you can get which can help fund enforcing justice.
|
| I believe you're looking at the moral right/wrong, and I
| don't believe that is the same right/wrong being
| discussed in terms of how lawyers often choose cases. At
| the end of the day, lawyers need work and they get that
| mostly through word of mouth and reputation. You don't
| really get either of those when you lose cases.
| XorNot wrote:
| The prosecutor doesn't see it that way. They see it as
| just "knowing" the guy is "definitely guilty". It's just
| like, a feeling you know? And a win will look great when
| they go for re-election (why is that even a thing?).
|
| Presuming rational actors in this case is missing the
| general problem with the system: people very easily
| convince themselves they know the truth _despite_ how the
| validity of the evidence changes. Whatever it said
| initially, that must be right - it 's misinformation 101.
| Once a belief is established it is much harder to change.
| nickodell wrote:
| >taking the life of an innocent
|
| The prosecutor isn't unilaterally deciding whether the
| DNA evidence is valid. There will be a public hearing
| where both the prosecution and defense show evidence
| about the validity of the DNA evidence, and a court will
| rule based on that evidence.
| bdavisx wrote:
| It would also give _every_ person convicted using their
| software an incentive to open an appeal.
| Spivak wrote:
| I like how this is considered a bad thing. Like we can't
| let this guy point out that he's being convicted by an
| unauditable black box that suddenly isn't worth using if
| it has to stand up to scrutiny because _then everyone
| would want to._ The horror.
|
| Like I'm actually kinda shocked this is the reality. I
| would have assumed that DNA evidence would have some
| blessed methodologies and tools/algorithms, with a strict
| definition of what constitutes a match or partial match
| specifically so this wouldn't happen.
| hluska wrote:
| This is one of these scary areas where reality matches my
| teenaged experiences playing Shadowrun. I used to hope
| that the brutal dystopia we played through was just fun.
| Now I'm seeing that the present needs a word even more
| brutal than dystopia. :(
| alchemism wrote:
| Kafkatopia
| knolan wrote:
| 170,000 lines of Matlab code for a project is not a good
| sign. Unless they're also including the source of various
| Matlab toolboxes which are already tested by the Mathworks.
|
| It's such a high-level language it's hard to imagine what the
| hell they're doing with all that code. It's probably mostly
| useless cruft from GUIDE.
| sdenton4 wrote:
| My guess is a bit of each: The company high-balling the LoC
| estimate to try to impress/scare the judge, but prooobably
| also has a truly terrible codebase.
| Aachen wrote:
| > If it would take 8.5 yrs to review, it's probably god
| awful, and should never ever ever be used to convict someone
| of such a crime.
|
| It's not like you review all scientific evidence and re-do
| the experiments that lead up to the discovery of <insert some
| evidence method> in the first place. Validating all that
| would also take years and much of it can be established as
| generally accepted by all parties. Similarly, there will be
| some trust involved with this source code as well. Getting
| the opportunity to look for bugs is essential in my opinion,
| but it needn't take multiple years. Focus on the parts you
| doubt, similar to what you'd do if you were reviewing the
| scientific method used in analog evidence.
|
| Of course, the two aren't identical. Validating scientific
| methods and validating a program is different in that the
| program is proprietary and the science (usually) merely
| behind a paywall. The latter can then be replicated by others
| and becomes established. The former will only ever be seen by
| that company and doesn't become established. So scrutiny is
| necessary, but after a couple cases that used an identical
| version, requiring access without articulating particular
| doubts would unduly delay the case. It doesn't seem
| unreasonable to start trusting the program after a bunch of
| defendants had experts look at it and found no way to cast
| doubt on its result. If you don't think software of 180k
| lines can be used in court under such circumstances because
| it would take too long to review, we should throw out pretty
| much all software anywhere in the judicial system. (That's
| not what you said, but some of the replies including yours
| hint at that.)
| vkou wrote:
| > It's not like you review all scientific evidence and re-
| do the experiments that lead up to the discovery of <insert
| some evidence method> in the first place. Validating all
| that would also take years and much of it can be
| established as generally accepted by all parties.
| Similarly, there will be some trust involved with this
| source code as well
|
| There are a few important differences between a generally
| accepted method, and some Matlab black-box that you feed an
| input into, and it prints out 'guilty' and 'not guilty'.
|
| 1. The former is based on centuries of peer review, where
| the best ideas eventually get selected for. The latter is
| an externally un-reviewed application, which encapsulates
| the best of whatever we could ship by Thursday.
|
| 2. You can call an expert witness to the stand, and ask
| them questions about the state of the art of <some evidence
| based method>. You can ask them why. You can ask them about
| how certain one should be about their statements. You can't
| cross-examine a black box.
|
| The actual solution to your quandary is to require that
| forensic analysis services must pass an annual,
| independent, double-blind analysis of the accuracy of their
| methods, before they are used in a courtroom - and that the
| results of those audits are made available to the defense.
|
| It's one thing for a man in a lab coat to take the
| microphone and say that their methods are accurate 'to
| within one in a million'. It's quite another to see an
| audit, where 100 samples were sent in for analysis over six
| weeks, and only 92 of them were analysed correctly.
|
| A jury might still convict on the basis of that 92%
| accuracy, but only if other meaningful evidence points
| against the defendant.
|
| Unfortunately, the reality of forensic science in 2021 is
| that most of it is sloppy bunk, with no assurances of
| accuracy.
| gowld wrote:
| In USA, accurate to 1 in a million means you can convict
| 300 innocent people for every guilty one.
|
| Bad stats, especially around DNA, has convicted many
| innocent people.
|
| BTW, law and Order did an episode on bad DNA science
| convicting someone.
| inetknght wrote:
| > _It 's not like you review all scientific evidence and
| re-do the experiments that lead up to the discovery of
| <insert some evidence method> in the first place._
|
| Actually, it _is_. That 's how science works and that's how
| convictions often get overturned.
|
| > _Validating all that would also take years_
|
| Are you suggesting that unvalidated data is being used to
| prosecute crimes?
|
| > _and much of it can be established as generally accepted
| by all parties._
|
| The point here is that it _isn 't_ established as generally
| accepted by all parties.
|
| > _Similarly, there will be some trust involved with this
| source code as well._
|
| "Trust but verify"
|
| > _If you don 't think software of 180k lines can be used
| in court under such circumstances because it would take too
| long to review, we should throw out pretty much all
| software anywhere in the judicial system._
|
| I firmly believe that if the source code isn't available to
| review by all parties, including the public, then it
| shouldn't be used in a criminal court.
| gowld wrote:
| > Are you suggesting that unvalidated data is being used
| to prosecute crimes?
|
| Yes. Pseudoscience is the bread and butter of criminal
| forensics.
| [deleted]
| maceurt wrote:
| It shouldn't take 10 lines an hour should it? I don't have
| experience reviewing professional code of this size, so please
| correct me if my assumption is wrong, but that number doesn't
| seem right.
| raldi wrote:
| I'd say more like ten lines a second, if you're skimming it
| to look for something. And of course, the work can be
| parallelized.
| novembermike wrote:
| It really isn't. Most of the code is probably going to be
| uninteresting and you can do 10 lines a minute or more. Some
| of the code will be more relevant and might take a day for 10
| lines. This would just be checking for accuracy though so you
| could probably just ignore a huge chunk of it.
| jjkaczor wrote:
| As part of a quality control team, I personally went through
| over 1.2 million lines of working code (i.e. not including
| comments) over the span of about 8 months, M-F, 9am-5pm.
|
| So, yeah - this number is bunk.
| wdobbels wrote:
| If their argument is that their codebase is basically
| unreadable, then I see why they're scared someone might find
| some bugs here and there.
| carlmr wrote:
| I'd get an expert witness in there to testify basically that.
|
| I also don't think you should code anything mission critical
| like this in Matlab. It's a decent language for prototyping,
| not for production.
| heavyset_go wrote:
| > _I 'd get an expert witness in there to testify basically
| that._
|
| Sounds really expensive.
| cyberlurker wrote:
| It might not be as much as you think. I know
| professionals who've been paid a few hundred dollars to
| be an expert witness more than once, but usually in
| medicine. It's easy money for someone if a lawyer often
| does certain cases that require an expert witness.
| Helloworldboy wrote:
| I'll do it for free
| ChrisLomont wrote:
| The numerics in Matlab are far better than pretty much any
| developer can produce in production. This is why Matlab is
| used in production - it's vastly more reliable than people
| rebuilding the things it is good at by hand for bespoke
| solutions.
| WmyEE0UsWAwC2i wrote:
| True. But I believe this is a case where correctness and
| clarity are the paramount concerns.
|
| There should be a public reference implantation of these
| methods if they are going to be used in court.
| kenjackson wrote:
| 8 years is a long time. What he then wanted to code review
| Matlab, and then the compiler that Matlab used, then do some
| silicon verification...
|
| Six to nine months seems like enough to do a very good code
| review with some testing. There's a good chance that 75% of
| that Matlab code doesn't execute for his test.
| edgyquant wrote:
| This is this persons life we're talking about. No amount of
| time is too much and it isn't hurting anyone to let him do a
| code review.
| kenjackson wrote:
| No. You give a reasonable amount of time. What if he asked
| for 100 years?
| edgyquant wrote:
| As long as he's actually reading the code, and still in
| jail unable to hurt anyone, let him have it. He'll likely
| pass before that time is reached and if he's just wasting
| time he'll get bored having to spend hours a day looking
| at code that he doesn't care to look at. I get what
| you're saying but my point is him wasting time reading
| code isn't hurting anyone and is a better use of an
| inmates time than sitting around.
| PurpleFoxy wrote:
| We should flip the responsibility. How does the company
| providing the software prove it is not flawed.
| abfan1127 wrote:
| 10 lines per hour? doesn't that seem painfully slow?
| flavius29663 wrote:
| have you ever seen MATLAB code?
| ashtonkem wrote:
| That's actually a pretty good argument for banning such
| code from the criminal justice system. The idea that
| unreadable code is deciding who gets locked up is really
| worrying.
| cozzyd wrote:
| and depending on which MATLAB version they run it in, it
| might have completely different results if they haven't
| tested carefully...
| deathlight wrote:
| This thread makes me sad because when I was taught and
| used matlab we had strong pressures to properly comment
| and document our code to make it legible (if only to our
| own future selves). It feels almost criminal to not do
| that in these circumstances.
| ehwhyreally wrote:
| Yes, You also dont read programs like a book, You generally
| follow the methods being used, Reading line to line would be
| like reading a a book with all its pages re arranged.
| techolic wrote:
| If it means what I think it means - understanding the code -
| sometimes it takes days to understand just one line of code.
| Document digging, googling, asking around, fiddling with test
| cases, reading production log etc.
| Aachen wrote:
| It depends on the level of scrutiny. It doesn't seem
| unreasonable. We review a lot more code per hour (usually
| C-like code though) but then we're not supposed to lock
| someone up for murder, we just find basic things like memory
| corruption. Don't even need to get into the business logic to
| find bugs that totally break the application, let alone all
| of it.
|
| When writing Python (I don't have stats about reading), a 1.0
| version of a small project took me 1.5 hours and consisted of
| 183 lines of code, so 2.2 lines per minute. That's much
| faster than this, but 183 lines is also a ton less complex
| than understanding the entirety of 180k lines and properly
| assessing whether it does exactly and only what it's supposed
| to.
|
| 10 lines per hour is probably taken as a lower bound to prove
| a point, especially because they argue about checking the
| whole thing (large parts can probably be skipped), but as a
| standalone statistic I would say it's probably within an
| order of magnitude from the true value. And for software time
| estimates that would be an amazing feat :p
| xirbeosbwo1234 wrote:
| Both 0.1 lines an hour and 1000 lines per hour would be
| equally wrong. That isn't how people would review that sort
| of code. They would test it and then thoroughly examine any
| areas of concern that crop up.
|
| I've run into 300-line programs that have taken me a month to
| figure out because the math was hard and I've run into
| 100,000 line programs that have taken me a few hours to tear
| apart.
| unishark wrote:
| A lot of bashing over a vague third-hand quote without a
| source.
|
| In the very next paragraph they say:
|
| > The company offered the defense access under tightly
| controlled conditions outlined in a non-disclosure agreement,
| which included accepting a $1m liability fine in the event code
| details leaked. But the defense team objected to the
| conditions, which they argued would hinder their evaluation and
| would deter any expert witness from participating.
|
| So it's a concern about IP protection for them.
| gowld wrote:
| There is no legitimate need for IP protection against
| destroying innocent lives.
| pnathan wrote:
| This is excellent! As a _principle_ , we should always be able to
| properly critique the expert witnesses and analyses! Or, at
| least, hire an expert firm to do the critique. It at least
| restores a proper ability to challenge an assertion of guilt.
| steerablesafe wrote:
| Are forensic labs often get blind tested? If there is a bias for
| guilty cases then it should turn out in those blind tests. Source
| code is a red herring here, there should be independent
| evaluations of forensic laboratories/methodologies/etc...
| regardless of software source code availability. Maybe these
| checks are already in place, I genuinely don't know.
| WaitWaitWha wrote:
| Not in my experience. Most labs of various types supposed to
| get certification, but these certifications primarily about
| chain of custody, operation protocols, record keeping, and
| such. It has little to do with the veracity of their
| conclusions.
| cmpb wrote:
| >Source code is a red herring here
|
| If there were a way to ensure that the test suite applied to
| these forensic labs was all-encompassing w.r.t. the genetic
| variables at play, then maybe. But that sounds impossible. What
| if there's a coding error that causes the software to operate
| differently/incorrectly only for people with a certain (rare)
| genetic abnormality?
|
| For what it's worth, I'm totally unversed in genetics, though I
| have a great deal of experience writing software tests (and
| seeing them come up short in adequately modelling real-world
| data).
| scott00 wrote:
| I don't know the answer to your question, but blind testing is
| complementary to (not a substitute for) source code review.
|
| It's very common for software to work correctly a high
| percentage of the time, but fail on rare input data. If, say,
| the software works correctly 999,999 times out of a million,
| you're going to be very unlikely to discover that error by
| throwing random samples at it, especially if you need a
| physical process (ie, drawing blood) in order to generate a
| test case.
|
| On the other hand, once you have a known failing case (as you
| would if the defendant knows the result must be in error
| because he didn't commit the crime), it's often fairly
| straightforward to identify the error by reviewing the source
| and/or using a debugger to examine the progress of the
| algorithm.
| fritzo wrote:
| Agreed, blind testing is important for statistical
| correctness and code review is important to avoid adversarial
| backdoors like dieselgate.
| lowbloodsugar wrote:
| "The co-founder of the company, Mark Perlin, is said to have
| argued against source code analysis by claiming that the program,
| consisting of 170,000 lines of MATLAB code, is so dense it would
| take eight and a half years to review at a rate of ten lines an
| hour."
|
| Well then the probability that it works correctly is zero.
| onetimemanytime wrote:
| Makes perfect sense. You have the right to cross-examine the
| witness, especially when you're being accused of murder.
| anfilt wrote:
| Link to the ruling:
| https://www.eff.org/files/2021/02/03/decision_appdiv_232021....
| femto113 wrote:
| Interesting that they're putting so much focus on the DNA piece,
| since the actual crime was witnessed by a police officer and
| there is circumstantial evidence tying the suspect to the weapon
| (it was found along the path they used to flee from the scene).
| vlovich123 wrote:
| I'm curious. I've always been frustrated by this "closed"
| business model in the legal system. I feel like the entire
| process & details should be detailed in the open (code,
| methodology, controls, etc). Of course the counterpoint is that
| it makes it easier to copy this business & undercut all the time
| & energy spent on building it (copying is easier). Is that the
| only reason? I feel like open kimono is a critically important
| concept for anything related to the legal system because of how
| any perversion removes its legitimacy. If it really is that
| prohibitive to run a profitable business in this space, is there
| open standards that can be enforced (e.g. "this is the core
| algorithm that is approved" & businesses must get regular audits
| to continue to be used & any failed audit causes a reexamining of
| any court cases you were involved in the past year?). That's less
| ideal because then who audits the auditors but maybe at least
| it's an acceptable middle ground from where we are?
|
| In general I've been extremely frustrated how regularly &
| consistently this entire industry keeps everything secretive &
| trust-based despite consistent examples of how insufficient trust
| is for this field & how devastating the results are when that
| trust is violated.
| dannykwells wrote:
| "170,000 lines of MATLAB code" (used as software to determine if
| humans are spending their lives in jail).
|
| Well if that's not a line to make you feel awful on a Monday
| morning.
| protomyth wrote:
| I do wonder if the next step is check the compiler / interpreter
| the code is running on. Is Matlab assumed to be error free in
| this situation? Will certified compilers and CPUs be required in
| the future? Looks like someone could end up reviving a modern
| version of the Viper
| https://en.wikipedia.org/wiki/VIPER_microprocessor
| aaron695 wrote:
| This isn't a DNA testing kit as one would normally think -
|
| "TrueAllele uses a hierarchical Bayesian probability model that
| adds genotype alleles, accounts for artifacts, and determines
| variance to explain STR data and derive parameter values and
| their uncertainty. The computer employs Markov chain Monte Carlo
| (MCMC) statistical sampling to solve the Bayesian equations. The
| resulting joint posterior probability provides marginal
| distributions for contributor genotypes, mixture weights, and
| other explanatory variables."
|
| https://onlinelibrary.wiley.com/doi/full/10.1111/1556-4029.1...
| Scandiravian wrote:
| How is that different from a DNA testing kit "as one would
| normally think"?
| trynton wrote:
| In such a serious case, once the accused was identified by
| TrueAllele, shouldn't a further test be carried out by a human to
| verify the results.
| Thorentis wrote:
| > If TrueAllele is found wanting, presumably that will not affect
| the dozen individuals said to have been exonerated by the
| software.
|
| Buried right at the end, but an interesting thought. It would
| depend on the individual cases, but if there was a whole body of
| evidence vs. one DNA test, surely these cases would need to be
| retried?
| clarle wrote:
| IANAL but if the individuals were already found not guilty, I
| think double jeopardy laws would apply.
|
| The prosecution wouldn't be able to try again unless they made
| an appeal that there's a substantially different crime.
| zyxzevn wrote:
| All source code for important decisions should be open and be
| verified by different parties. Just as that for voting machines.
| [deleted]
| retrac wrote:
| There's a certain irony in advocating for capital punishment on
| a thread about the dangers of lying prosecutors, bogus
| forensics, and false convictions.
| AS37 wrote:
| I'd like to encourage you, 5 minutes after you learn about
| someone's existence, _not_ to wish for their death, execution,
| imprisonment, and /or disenfranchisement. The real world is
| too complicated to judge correctly in that time.
|
| As well, please remember that the comment guidelines exist at
| https://news.ycombinator.com/newsguidelines.html
| [deleted]
| [deleted]
| dang wrote:
| Thanks for pointing this out! We've deleted that comment by
| the commenter's request.
| [deleted]
| dang wrote:
| We detached this subthread from
| https://news.ycombinator.com/item?id=26064707.
| fritzo wrote:
| Code review is not the way to validate statistical inferences.
|
| Even during internal review of statistical inference code,
| committers argue correctness through a combination of readable
| code, readable tests, and statistical validation on a combination
| of synthetic and real data. While I agree the TrueAllele source
| should be provided, the "zero-defects" bar is neither sufficient
| nor necessary for correctness of the inferences made.
| akyu wrote:
| > Code review is not the way to validate statistical
| inferences.
|
| But it is a way to invalidate the inference, which is what the
| defense is interested in.
| damagednoob wrote:
| How much uncertainty in the system are you willing to live with
| if you were falsely accused of murder?
| fritzo wrote:
| That is a question for the judge or jury. The statistician's
| job is to provide a probability distribution, and to argue
| for the statistical correctness of that distribution based on
| similar inferences made on validation data.
| damagednoob wrote:
| In that case, at least with respect to the death penalty,
| "zero-defects" is precisely what the courts have moved
| towards.
| hirundo wrote:
| If an accused person has the right to see the source code that
| produced evidence against them, is it a violation of their rights
| for the source code to be obfuscated, or even just so
| spaghettified that not even an expert can understand it?
|
| I kinda think that should be a violation. But deciding whether a
| particular piece of code is so bad is so subjective that I'm not
| sure on how you'd make a legal standard out of it. Maybe start
| with "the linter found a ratio of warnings to lines > X%" or some
| such.
|
| Having a legal standard of code coherence/incoherence might help
| filter pull requests. "This PR cannot be merged to this project
| because it is configured to reject legally incoherent code."
|
| As code becomes more complex it may become more meaningful to
| have access to the test suite, and to challenge the evidence if
| the tests are inadequate to demonstrate the correct code
| behavior.
| Jakobeha wrote:
| In the case where code is used to convict or acquit someone, I
| think it should be a well-tested and established program:
| generally something with the software quality of Linux, or in
| this case, whatever DNA testing kit is being used by scientists
| in top-ranking universities.
|
| We could also use formal verification based on well-established
| axioms. For example, maybe we could "prove" that the DNA kit
| reports accurate results as long as the samples it's given are
| processed correctly.
| londons_explore wrote:
| You don't need a legal standard. You just need to put doubt
| into the minds of a jury. You can get an expert to stand up and
| say "I'm an expert in computers, and I couldn't understand how
| this DNA test works. I think it's likely there are mistakes in
| it that neither I nor the people that made it have discovered".
| Spivak wrote:
| "This system is an absolute mess that is impossible audit and
| so there is no way that the company using could find bugs in
| it either."
| [deleted]
| WaitWaitWha wrote:
| During discovery, the opposing lawyer can raise an alarm to the
| court of such obfuscation. There have been many cases where
| such behavior cost cases going against the party.
___________________________________________________________________
(page generated 2021-02-08 23:00 UTC)