[HN Gopher] Interviewing my mother, a mainframe COBOL programmer...
___________________________________________________________________
Interviewing my mother, a mainframe COBOL programmer (2016)
Author : MoBarouma
Score : 214 points
Date : 2023-12-14 18:34 UTC (4 hours ago)
(HTM) web link (ezali.substack.com)
(TXT) w3m dump (ezali.substack.com)
| scrlk wrote:
| The original HN submission from 2016:
| https://news.ycombinator.com/item?id=12096250
|
| > "The banking programming world is a completely different world
| than what most of us are used to"
|
| If you're after another read on this topic, "An oral history of
| Bank Python" is good: https://calpaterson.com/bank-python.html
| (also previously on HN:
| https://news.ycombinator.com/item?id=29104047)
| SilasX wrote:
| Oh wow, I can't believe that was seven years ago! I never got
| around to interviewing my mom about programming mainframes with
| Michigan Algorithm Decoder:
|
| https://news.ycombinator.com/item?id=12097032
| otteromkram wrote:
| This is great! I wonder how the update(s) went since the article
| was written in 2016 and they mentioned 4 years per country, or 16
| years total. COVID-19 probably threw a wrench in that plan, but
| how big of an impact is another interesting question.
| vanderZwan wrote:
| > _This position is the most important one in the bank, at least
| from a technical standpoint. If, let's say, my mother and
| everyone on her team would quit their job, the bank would go
| under within a matter of weeks if they're lucky._
|
| And given how big the market share of Nordea is in Sweden (and
| other Nordic countries, for that matter) that would probably
| bring down the Swedish, and possibly Nordic economy. Which would
| then impact a lot of the EU as well, I guess.
|
| Ever since reading this article I've wondered if these COBOL
| programmers that keep banks like this running are an enormously
| underestimated "bus factor" for many of the world's economies,
| and what kind of back-up plans they have for such a scenario.
|
| [0] https://en.wikipedia.org/wiki/Nordea
| TomK32 wrote:
| As a programmer who has been hit by a bus (after riding on that
| same bus for five hours to it's endstop in western Greece) I'd
| like to point out that getting hit and killed by a car is a
| much higher risk. Add the rise of SUV sales which are more
| dangerous to pedestrians and cyclists, and you might be as
| surprised as I am that banks don't go down due to dead
| programmers but due to bad accounting and fraud...
| rsynnott wrote:
| > and what kind of back-up plans they have for such a scenario.
|
| I mean, notoriously, stop working, for weeks:
| https://en.wikipedia.org/wiki/TSB_Bank_(United_Kingdom)#Migr...
|
| (I've got to assume that loss of institutional knowledge was a
| big factor in that fiasco.)
| brightball wrote:
| .
| rajamaka wrote:
| Very creepy. I was talking about programming and alas I com to
| HN and topics related to programming are ob the front page!
| brightball wrote:
| And yet, you don't see much about mainframes and COBOL on
| here ever...nor do I often have lunch conversations about it.
|
| The timing was just interesting.
| pavel_lishin wrote:
| I wish I had talked to my grandmother more before she developed
| dementia. I knew she was a mathematician and programmer, but
| briefly speaking to her a few years back, she mentioned that one
| of her jobs was calculating orbits for satellites in the Soviet
| Union.
|
| At least, I hope they were satellites!
| ca_tech wrote:
| I would highly encourage anyone who has even thought about
| doing this; do it. If you are thinking about how to get
| started, I recommend the StoryCorp app. It is easy to use they
| have a bunch of prebuilt questions or you can create your own.
| You record directly to your phone but if you are inclined, you
| can upload your interview to the Library of Congress.
| https://storycorps.org/participate/storycorps-app/
| artemavv wrote:
| There is also a project dedicated to saving the wisdom of
| earlier generations: https://savewisdom.org/the-1000-word-
| save-wisdom-questions/
| CapricornNoble wrote:
| I'm in a similar boat. My mother worked with software
| simulations for the AEGIS system. Not very common work for
| black women in the 1980s. She earned her mathematics degree in
| 3 years, graduating at 20, and used to program computers with
| punch cards in the 70s.
|
| Now she can't tie her shoes consistently.
|
| To everyone reading, capture the stories of your loved ones
| life accomplishments before it's too late!
| Tor3 wrote:
| This. Not related to the main story here - I'm the first
| programmer in my family - but my great-grandfather was born
| in 1865 and lived a long life, he was blind for his last 30
| or 40 years but had a phenomenal memory. A neighbor had the
| insight to use one of the very first tape recorders, just
| after WW2, together with my father who was a boy at that
| time, to record my great-grandfather's stories about what he
| did in life. When my father retired he cobbled together a
| working tape recorder from two or three old ones, and copied
| the stories over to cassette, and later to CD. I listened to
| one of those recordings the other day - my great-grandfather
| told about how when he as a young boy, approximately around
| 1872-1873, he met a very old man who told about how he had
| learned the origin of the name of where they (he and my
| great-grandfather) lived, from another man when _he_ was a
| boy. That origin turned out to be very different from what
| everyone these days are guessing, but immensely more
| plausible. Brought to me from the 18th century, via my great-
| grandfather.
| rsynnott wrote:
| > At least, I hope they were satellites!
|
| Could be both:
| https://en.wikipedia.org/wiki/Fractional_Orbital_Bombardment...
| guyzero wrote:
| Programming a mainframe while working from a hotdesk. Who would
| have imagined.
| hyggetrold wrote:
| I'm very curious about IMS - a mentor described it as a graph
| database of sorts back in its day. I was told it was a great
| system (at least for its time). Does anyone have a perspective?
| epc wrote:
| Hierarchical, not relational, not what I'd call a graph
| database. Pre-SQL.
|
| https://en.wikipedia.org/wiki/IBM_Information_Management_Sys...
| hyggetrold wrote:
| Yeah I think what they actually called it was "network
| database" which I thought maybe was like a graph.
| shortsightedsid wrote:
| > What's your working environment like? - We've recently moved to
| a more "hip" location. We used to have personal desks, but now we
| have this "pick whatever spot is avaiable" open area. I dislike
| it a lot.
|
| Somehow this resonates a lot with me even if though I've never
| worked on Mainframes or anything like that.
| artemavv wrote:
| I, too, strongly prefer having a personal desk. It is
| completely natural for a human to set up his environment
| according to their tastes and preferences, and I'm baffled that
| some office designers do not account for that.
| ptmcc wrote:
| > I can only imagine the fat paycheck a 20-year old mainframe
| programmer would get though, because your age in this case would
| be invaluable.
|
| It's funny, people frequently assume this would be true but
| reality doesn't really bear this out. It's typically pretty
| average to even below average which contributes to the talent
| pipeline problem.
|
| The other side of it is that its not the technical stuff like
| "knows COBOL" that is so immensely valuble. Any average dev can
| "learn COBOL" but that's not actually the valuable thing. The
| anecdotes of COBOL programmers coming out of retirement for
| 500k/yr contracts has little to do with COBOL, but their
| accumulated institutional knowledge of the giant ball of business
| logic encoded in that COBOL.
|
| If these banks and other institutions actually did write fat
| paychecks to young mainframe programmers the demographic problem
| they're facing might not be so bad.
| Deprecate9151 wrote:
| I used to work at a large insurance company with a COBOL core
| system. I completely agree with this. They paid for people who
| knew all the undocumented idiosyncrasies and foot-guns in the
| code. Not just "Knowing COBOL". A decent programmer could read
| a COBOL program and follow it.
|
| They wouldn't know that the reason the key export fell over
| when extracting data for your 10K filing because someone had
| decided/assumed/whatever that a certain record count would
| never go over 3000, so they hard coded the program to just
| error out if it went above that value.
| ecshafer wrote:
| When I was working in a finance company that used cobol, sure
| I could read cobol. I could probably write some cobol. But
| the extra crap around cobol, how mainframes work, DB2, JCL,
| etc. is where this gets complicated.
| importantbrian wrote:
| I went to grad school with a guy who was in that situation. He
| worked at a bank and got laid off during the financial crisis.
| The plan was for the bank to port the old system over to Java
| or something like that, and they were "close" to the end of the
| project and they thought at that point the rewrite team was
| comfortable enough with COBOL that they could do the rest of
| the rewrite without him. Turns out that yes they understood
| COBOL just fine, but they desperately needed his institutional
| knowledge. He ended up agreeing to come back on a part-time
| basis at some obscene consulting rate for however long the
| transition took and in the meantime, he did grad school part-
| time to skill up.
| DaveSchmindel wrote:
| I can vouch that this is true in some cases.
|
| I worked for a multinational bank headquartered in the U.S.,
| whose corporate team came up with a sexy program to snatch up
| recent Computer Science graduates. The program offered them a
| "fair" market rate for an entry-level jobs in industry, and let
| the graduates try two to three different teams and/or
| departments in their first year. At the end, if the company
| still liked them, the candidate got to pick their permanent
| team for a full-time position. This position, however, did not
| see a bump in pay, and by that time the program effectively
| filtered out candidates that weren't performing at an
| accelerated level.
|
| The results were great for the business: young programmers that
| were often more proficient in COBOL than their new peers were
| costing them $60k-$70k a year. The senior, or "tenured," peers
| were in the $200k-$400k range in some cases.
| tristor wrote:
| > If these banks and other institutions actually did write fat
| paychecks to young mainframe programmers the demographic
| problem they're facing might not be so bad.
|
| When I first started work I was involved in banking/insurance
| mainframe stuff, and it was honestly a pretty terrible working
| environment for reasons that spanned from the mundane (like
| dress codes) to the esoteric (like the horrible crufty legacy
| codebase). I would have put up with it if it paid well, but it
| didn't. In fact, it paid significantly worse than the job I'd
| had before which was installing physical cable plant (fiber
| optics and copper ethernet), the reason I took it was at least
| the office had air conditioning, but it certainly wasn't
| something I wanted to turn into a career.
|
| As a mid-level windows sysadmin doing customer-facing phone
| support at a cloud provider, I made nearly 30% more than I had
| as a junior mainframe guy at a bank/insurance company. The
| difference in the skillsets and their commonality was massive,
| yet the pay was significantly worse for the mainframe work
| despite it being a rare skillset in a high value industry. I
| remembered when I was in this job that guys who worked on
| trading desk backend code were far better compensated while
| working in easier development environments on less esoteric
| codebases, the mainframe folks were the lowest paid of the
| developers at that company, at every level of seniority. The
| only person we worked with who was well compensated was an
| independent contractor who'd worked there previously for 30
| years before retiring.
| beretguy wrote:
| Last time I checked DMV of SC pays $50K salary with no remote
| work option for COBOL position. And requires 2 years of
| experience. Granted, it's a state job, I get it, but still.
| uudecoded wrote:
| It's probably because they technically have to make that job
| posting and let it sit before handing it off to a contract
| firm for $300k.
| jongjong wrote:
| In tech, it's only true if you know some dirty secrets. The
| more dirt you are exposed to, the more you keep your mouth
| shut, the more it pays. The thing is that teams are so large
| and compartmentalized that nobody knows the true horror of
| what's going on.
| thaumaturgy wrote:
| Yep. I have COBOL experience (on a mainframe even!), and I'd
| even be willing to do it again -- and I'm young by the
| standards of COBOL programmers, so somebody could even get 20
| years out of me still.
|
| But the wages being offered for those roles are abysmal
| relative to what's available for other skills, and modern life
| is applying a great deal of pressure to chase larger paychecks,
| sadly.
| zengid wrote:
| <Anecdote> My first programming job was at a transportation
| enterprise with many programmers that had been working there
| 15-20+ years, many of whom mainly maintained the COBOL that ran
| the business. Many of those senior programmers were women; it
| felt like over 50% in the senior cohort. What really made me sad
| was that the younger programmers there were predominantly male
| (although still quite a few female!). That company mainly hired
| out of the local engineering colleges, so it was an interesting
| case study on how the number of women entering software
| engineering programs went down over the years. </Anecdote>
| duderific wrote:
| I recall hearing (can't remember where) that in the olden days,
| computer programming was considered akin to typing or
| secretarial work, so at first it attracted mostly women.
| danielodievich wrote:
| My grandmother programmed with punch cards, have no idea on what
| hardware and it's too late to ask. My father did a bunch of
| fortran and Cobol on USSR mainframes and then did a bunch of y2k
| here in USA. One of the neatest things I have from this is a
| printout from one of his fortran programs from I think Minsk-32
| mainframe which he ripped into 3 pieces to wrap a developed large
| format 64mm film of some mountains that he shot a long time ago.
| The program seems to be called MATR1 and is doing matrix
| manipulations and referring to topography of the land in
| comments. I have it framed on my wall near my workstation. I
| coded in variety of languages most of my life And now my teenager
| looks to be interested in coding and is doing Java and Python in
| high school. Here is to 4th generation programmer!
| 999900000999 wrote:
| Sounds like a neat movie idea.
|
| The same code base passed down though generations.
|
| Legacy code.
| jacobyoder wrote:
| The code will only run on some specific soviet-era hardware,
| and the last hardware was destroyed 20 years ago. There's
| some new regime that is rumored to have rebuilt the original
| hardware, and is now on the hunt for the missing source code
| that is framed above the great grandchild of the original
| developer, who was executed for treason. The new bad guys are
| gunning for the source code, and it has to be protected at
| all costs, to prevent a takeover of the world.
|
| I'm already seeing the inheritance jokes write themselves.
|
| Someone have chatgpt write the screenplay now.
| micah94 wrote:
| I think you just wrote the prompt right there. Step 2. ????
| Step 3. PROFIT!
| ikesau wrote:
| Class Warfare, out 1717243201.
| hnthrowaway0315 wrote:
| 1TB transaction per year seems pretty small data. I guess IBM
| stuffs are there for the stability then?
| awestroke wrote:
| The IBM stuff is pure legacy. Of course this would all perform
| much better with modern tools, like Postgres.
| dan-robertson wrote:
| Using DB/2 surely makes a lot of sense for integration with the
| rest of their IBM tech but also something the new system could
| presumably talk to too. I would guess that it performs ok at
| the kind of 'single big OLTP database server' tasks it was
| meant for. Notably IBM allows you to publish benchmarks for
| their databases (so long as you provide detailed methodology
| and take steps to tune it correctly) whereas Oracle and
| SQLServer do not allow publishing benchmarks without
| permission. Though one could argue this is just IBM feeling
| like benchmark performance wouldn't matter much to their
| customers.
| dfee wrote:
| (meta) As a platform, substack content must follow the power law,
| right? That is, a few authors are most productive and there's a
| long tail of single (or zero) article substacks.
|
| This article was migrated from medium, but now the author gets to
| collect email addresses. But why subscribe? There are two
| articles. Is substack an RSS replacement (potentially with a
| paywall)? Maybe I should just consider substack to be a low
| barrier to entry blogging platform that can be set to your
| personal domain for $50?
|
| Definitely not trying to beat up on the offer, just curious about
| the real value prop of substack for "most" producers and if
| that's aligned with my goals as a reader who sees a email
| collection modal on every visit.
| spelunker wrote:
| My mother-in-law worked her entire career in an IT department of
| an insurance company. She never really did much programming, but
| became a source of domain knowledge over time.
|
| She has all kinds of amusing stories of ye olden days. Before a
| proper computer system, everything was stored in physical
| documents of course, and her first project out of college was
| traveling to satellite offices to re-organize the file systems
| there to match the new strategy that HQ had come up with. The
| business of satellite office would grind to a halt while a team
| of people literally took every single document out, re-labeled
| it, and put it back. The whole project took over a year.
|
| That gave some perspective about my job, lol.
| dgadj38998 wrote:
| > ISPF is directly connected to the mainframe, and there's no
| such thing as a local development environment here.
|
| That's pretty crazy
|
| Sounds like having the whole team SSH into one server and doing
| all the work through the terminal
|
| I'm imagining editing my co-workers files and just removing a
| random semicolon to mess with them
| flyinghamster wrote:
| That was the 1960s-80s way of working with computers. Multiple
| terminals connected to a single timesharing mainframe (or mini,
| for that matter). You _might_ have a network like DECNET, but
| mostly it would be hardwired terminals, or if you were less
| lucky, a dialup modem and an acoustic coupler at a blazing 300
| bps. You were sharing one machine with potentially dozens of
| other developers.
|
| Legacies of this abound in Linux (even the ability to SSH in is
| a descendant of this world). Commands like "who" are there to
| show who's logged in, and there's even an entire process
| accounting system that can be switched on to bill your users
| for CPU time.
| dgadj38998 wrote:
| Was there a way to message other people who are logged in?
| Like a slack equivalent?
| flyinghamster wrote:
| On Unix, there was the "talk" command for split-screen
| chat, and also "write" to just send a message to someone
| else's terminal subject to permission with the "mesg"
| command. Not all systems had such provisions (or, in come
| cases, they had to be written by users).
| ianmcgowan wrote:
| talk, or later ntalk in Unix. Or wall to send to everyone.
| I used a system in the 80's (Pick DB) that had a send-
| message command. Also tandem which allowed you to take over
| someone's terminal like screen-sharing now. If you were the
| super-user you could do it without asking for permission,
| and start typing messages to the unlucky user..
| bennysaurus wrote:
| Things is source code management these days even in mainframe
| but many shops still have you log into the same "box" to do
| your development. It can get pretty fun when trying to
| coordinate testing and patches.
| jll29 wrote:
| Thanks for sharing the story of your developer mom - a very cool
| mother to have!
|
| COBOL is not a "cool" language, but mainframes have been around
| long enough to be "retro cool" now, and most run Linux at least
| as an optional OS under some virtualization (IBM Z).
|
| As a doctoral candidate in the noughties, I purchased a book
| about FORTRAN due to its "retro-coolness" and read it, and
| eventually took on a short university gig to earn the money that
| the book cost back, tutoring architects/engineers in FORTRAN 95
| for a bit, which was fun; to date, I could not bring myself to do
| the same with COBOL, though, or not yet.
|
| Because it is so verbose, if I had to use it I would probably
| write in another language and transpile to it.
| flyinghamster wrote:
| I've probably touched on this before, but back in the 1980s,
| absolutely none of the computer courses I took ever even so
| much as touched an IBM mainframe. I almost got the impression
| that there was an effort _not_ to teach them.
|
| On the other hand, you could expect to learn a new operating
| system every year. Oh, we don't use DEC gear at this school,
| check out our spiffy Prime running PRIMOS. Lather, rinse,
| repeat.
| syngrog66 wrote:
| the 1st professional programmer I knew was a woman who did COBOL
| for a bank
|
| its struck me as interesting ever since. because over the course
| of my life and career since its seemed that 99%+ of the
| programmers I knew of were male.
|
| I don't think there is any one right/ideal breakdown by gender,
| so I just like to have best sense for facts on the ground
| dan-robertson wrote:
| I think one thing is that the programmer job somewhat evolved
| from data entry and clerical roles that employed plenty of
| women already. So in some sense it didn't start as a 'man's
| job'. For slightly later generations of programmers, one can
| observe that computer science degrees peaked near 40% female in
| the mid '80s in the US[1]. It's closer to 20% now and there are
| much more graduates today so you should probably expect a
| proportion somewhere around 20% of working CS grads. That
| number doesn't seem crazily unrealistic to me for working
| programmers, though obviously it varies between companies. <1%
| does seem unrealistic to me. I'm slightly surprised that that's
| been your experience (I assume you aren't exaggerating because
| you want a sense for facts on the ground).
|
| [1] eg https://www.gcu.edu/blog/gcu-experience/analysis-women-
| compu...
| derefr wrote:
| > Banking systems are also extremely advanced. A personal bank
| account differs a lot from a business bank account, and there are
| at least 50 different types of bank accounts for each of them.
|
| I wouldn't necessarily call that "advanced"... maybe more,
| lacking in requirements-analysis-time insight in ways to factor
| the business-domain into HAS-A component relationships, such that
| individual components and their ADTs can be shared across parent
| types [which are really just template/factory objects for a
| smaller set of actual types], and initialized with simple
| parameters that get used as formula variables with no piecewise
| logic.
|
| To be clear, I write this as someone who works for a company that
| maintains a unified representation of data across the blockchain
| ecosystem -- where each blockchain has its own peculiarities
| about what an "account" is, what a "transaction" can do, etc. Our
| data model only has _one_ toplevel account type, _one_ toplevel
| ledger-transaction type, etc. To handle the peculiarities, our
| data model instead has a large heirarchy of smaller data-objects
| hanging off of those toplevel ones; where any given data-object
| is sort of an "optional extension" that may or may not be there
| depending on how the toplevel object was created.
|
| This approach allows us to just have one unified _code-path_ that
| treats every account like every other account, every tx like
| every other tx, etc. We don 't have duplicate code or a hierarchy
| of subclasses that all do things slightly differently; but
| instead, for _any_ ledger-transaction, there may or may not be
| e.g. a strategy-pattern object hanging off that tx -- and if
| there is, it gets used instead of the default static one. It 's
| great for maintainability, testability, predictability,
| cacheability, and hundreds of other things.
|
| I'd love to know if there's any _good_ reason that a bank would
| actually _want_ "50 different types of bank account" on an
| implementation level, rather than these all boiling down to one
| type with varying values of certain state variables +
| presence/absence of certain foreign-key relationships.
|
| Other than maybe "some of these account types are actually a part
| of completely different data models living in third-party
| systems, that we acquired, and then never merged into our own
| systems." ;)
| boricj wrote:
| > I'd love to know if there's any good reason that a bank would
| actually want "50 different types of bank account" on an
| implementation level.
|
| Not having the benefit of hindsight?
|
| Some banks are hundreds of years old. Most of them were
| computerized seventy years ago if not earlier, back in the
| stone age of computers. These kinds of banks won't bet the
| house on a newfangled system every couple of years because some
| bright-eyed engineer told them it's the trend nowadays.
|
| Not messing up their bookkeeping is their number one priority.
| People would riot if their bank told them "sorry, we no longer
| know how much money you had deposited with us".
| spidermonkey23 wrote:
| My company which uses mainframe and COBOL is undergoing a
| migration to Linux. This involves porting all the batch command
| scripts to bash and also fix quirks with COBOL compiler
| differences. At the end of it though the system is at least 4x
| faster and much more maintainable. This was the solution instead
| of doing a complete rewrite to Java that they backed out of some
| years before.
| bennysaurus wrote:
| Do you know what they're using? Are they going to Microficus
| COBOL or doing something different?
| ape4 wrote:
| I surprised banks don't have a domain-specific language (DSL) to
| describe their businesses. Instead each bank has to code "savings
| account" in COBOL.
| dan-robertson wrote:
| What domain do you think COBOL was made for?
|
| Basically the only things computers were worth it for back then
| were scientific calculations and bookkeeping. Banking is
| obviously on the more complicated end of bookkeeping. Insurance
| did get a special COBOL variant because I guess they had some
| tasks shaped more like scientific calculations too.
| boilerupnc wrote:
| Recently announced, there's also now a generative AI tool to
| assist developers with their COBOL modernization journey to Java
| [0]. I've heard pretty decent reviews on its effectiveness.
|
| [0]. https://www.ibm.com/products/watsonx-code-assistant-z
___________________________________________________________________
(page generated 2023-12-14 23:00 UTC)