diff --git a/content/thoughts/things-i-learnt-the-hard-way.md b/content/thoughts/things-i-learnt-the-hard-way.md new file mode 100644 index 0000000..011ebd8 --- /dev/null +++ b/content/thoughts/things-i-learnt-the-hard-way.md @@ -0,0 +1,687 @@ ++++ +title = "Thing I Learnt The Hard Way (in 30 Years of Software Development)" +date = 2019-06-10 + +[taxonomies] +tags = ["en-au", "programming", "work"] ++++ + +This is a cynical, clinical collection of things I learnt in 30 years working +with sofware development. + +Again, some things are really cynical, others are long observations on +different jobs. + + + +## Software Development + +### Spec first, then code + +If you don't what you're trying to solve, you don't know what to code. + +Write something specifying how the application works before writing any code. + +"Without requirements or design, programming is the art of adding bugs to an +empty text file." -- Louis Srygley + +Sometimes, even an "elevator pitch" -- up to two paragraphs that describe what +the application does -- is enough. + +The times I stood longer looking at my own code wondering what to do next were +when we didn't have the next step defined. It is a good sign that it's time to +stop and discuss it with your coworkers -- or maybe rethink the solution. + +### Write steps as comments + +If you have no idea how to start, describe the flow of the application in high +level, pure English/your language first. Then fill the spaces between comments +with the code. + +Better yet: think of every comment as a function, then write the function that +does exactly that. + +### Gherkin is your friend to understand expectations + +Gherkin is a test description format which points "Given that the system is in +a certain state, When something happens, then this is expected". Even if +you don't use any testing tool that reads Gherkin, it will give you a good +understanding of what it is expected from the app. + +### Unit tests are good, integration tests are gooder + +On my current job, we do test modules and classes only (for example, we write +tests for the view layer only). It gives us some idea if things are going right +or not, but they lack a view of how the whole is going on -- a thing +integration tests, which tests how the system as a whole behaves -- do better. + +### Tests make better APIs + +We code in layers: There is the storage layer, which should make our data +permanent; there is a processing layer, which should do some transformation on +the data stored; there is a view layer, which has information on how the data +must be present; and so on. + +As I mentioned, integration tests feel better, but testing layers by themselves +can give you a better view on how their API looks like. Then you can have a +better look on how to call things: Is the API too complex? Do you have to keep +to much data around to be able to make a single call? + +### Good languages come with integrated tests + +You can be sure that if a language brings a testing framework -- even minimal +-- in its standard library, the ecosystem around it will have better tests +than a language that doesn't carry a testing framework, no matter how good the +external testing frameworks for the language are. + +### Future thinking is future trashing + +When developers try to solve a problem, they sometimes try to find a way that +will solve all the problems, including the ones that may appear in the future. + +But here is the thing: The problems from the future will never come and you'll +end up either having to maintain a huge behemot of code that will never be +fully used or you'll end up rewriting the whole thing 'cause there is a shitton +of unused stuff. + +Solve the problem you have right now. Then solve the next one. And the next +one. At one point, you'll realize there is a pattern emerging from those +solutions and _then_ you'll find your "solve everything". + +### Documentation is a love letter to your future self + +We all know writing the damn docs for functions and classes and modules is a +pain in the backside. But realizing what you were thinking when you wrote the +function will save your butt in the future. + +### The function documentation is its contract + +When you start the code by writing the documentation, you're actually making a +contract (probably with your future self): I'm saying this function does _this_ +and _this_ is what it does. + +If later you find out that the code doesn't match the documentation, you have a +code problem, not a documentation problem. + +### If a function description includes an "and", it's wrong + +Functions should do one thing and one thing only. When you're writing the +function documentation and find that you added an "and", it means the function +is doing more than one thing. Break that function into two and remove the +"and". + +### Good languages come with integration documentation + +If the langauge comes with its own way of documentating +functions/classes/modules/whatever and it comes even with the simplest doc +generator, you can be sure that all the language +functions/classes/modules/libraries/frameworks will have a good documentation +(not great, but at least good). + +Languages that do not have integrated documentation will, most of the time, +have a bad documentation. + +### A language is much more than a language + +A programming language is that thing that you write and make things "go". But +it has much more beyond special words: It has a build system, it has a +dependency control system, it has a way of making tools/libraries/frameworks +interact, it has a community, it has a way of dealing with people. + +Don't pick languages just 'cause they easier to use. + +### Sometimes, it's better to let the application crash than do nothing + +Although that sounds weird, it's better to not add any error handling than +silently capturing errors and doing nothing. + +A sadly common pattern in Java is + +```java +try { + something_that_can_raise_exception() +} catch (Exception ex) { + System.out.println(ex); +} +``` + +This does nothing to deal with the exception -- besides printing it, that is. + +If you don't know how to handle it, let it happen,so you can figure out _when_ +it will happen. + +### If you know how to handle the issue, handle + +Counter-point to the previous point: If you know when something will raise an +exception/error/result and you know how to handle it, handle it. Show an error +message, try to save the data somewhere else, capture the user input in a log +file to later processing, but _handle_ it. + +### Types say what you data means + +Memory is just a sequence of bytes; bytes are just numbers from 0 to 255; what +those numbers mean is described on the language type system. + +For example, in C, a `char` type of value 65 is most probably the letter "A", +which an `int` of value is 65 is the number 65. + +Remember this when dealing with your data. + +This is what most people get wrong about adding booleans to check the number +of True values. Here, let me show you an example of JavaScript that I saw +recently: + +```javascript +console.log(true+true === 2); +> true +console.log(true === 1); +> false +``` + +### Understand and stay way of cargo cult + +"Cargo cult" is the idea that, if someone else did, so can we. Most of the +time, cargo cult is simply an "easy way out" of a problem: Why would we think +about how to properly store our users if X did that? + +"If BigCompany stores data like this, so can we". + +"If BigCompany is behind this, this is good." + +### "Right tool for the job" is just to push an agenda + +"Right tool for the job" should be an expression that meant that there is a +right and a wrong tool to do something -- e.g., using a certain +language/framework instead of the current language/framework. + +But every time I heard someone mention it, they were trying to push their +favourite language/framework instead of, say, the right language/framework. + +### "The right tool" is more obvious than you think + +Maybe you're in a project that needs to process some text. Maybe you're +tempted to say "Let's use Perl" 'cause you know that Perl is very strong in +processing text. + +What you're missing: You're working on a C shop. Everybody knows C, not Perl. + +Sure, if it is a small, "on the corner" kind of project, it's fine to be in +Perl; if it is important for the company, it's better that if it is a C +project. + +PS: Your hero project (more about it later in this doc) may fail due this. + +### Data flows beat patterns + +(This is personal opinion) When you understand how the data must flow in your +code, you'll end up with better code than if you applied a bunch of design +patterns. + +### Design patterns are used to describe solutions, not to find them + +(Again, personal opinion) Most of the time I saw design patterns being +applied, they were applied as a way to find a solution, so you end up twisting +a solution -- and, sometimes, the problem it self -- to fit the pattern. + +First, solve your problem; find a good solution; then you can check the +patterns to know how you name that solution. + +I saw this happens _a lot_: We have this problem; a design pattern gets close +to the proper solution; let's use the design pattern; now we need to add a lot +of things around the proper solution to make it fit the pattern. + +### Cognitive Dissonance is the readability killer + +"Cognitive dissonance" is a fancy way of saying "I need to remember two (or +more) different things at the same time to understand this." + +For example, adding booleans to count the number of True values is a mild +cognitive dissonance; if you're reading a piece of code and see a `sum()` +function, which you know makes the sum of all numbers in a list, you'd expect +the list to be composed of numbers, but I've seen people using `sum()` to +count number of True values in a list of booleans, which is confusing as heck. + +### The Magical Number Seven, Plus or Minus Two + +"The magical number" is a psychology article about the number of things one +can keep in their mind at the same time. + +If you have a function, that calls a function, that calls a function, that +calls a function, that calls a function, that calls function, you may be sure +it will be a hell to read later. + +Think more about: I'll get the result of this function, then pass it to the +second function, get its result, pass to the third an so on. + +But: + +1. Today, psychologists talk more about the magical number FOUR, not seven. +2. Think function composition (as in "I'll call that function, then that + function, then that function..."), not function calling (as in "That + function will call that function, that will call that function..."). + +### Shortcuts are nice, but only in the short run + +A lot of languages/libraries/frameworks add a way to make things shorter, +reducing the number of things you need to type. + +But, later, that will bite you and you'll have to remove the shortcut and do +the long things. + +So learn what the shortcut does before using it. + +You don't need to write things the hard way first and then clean up using the +shortcuts: All you need to do is what the shortcut does in the background, so +you at least have knowledge of what can go wrong using it, or how to replace +it with the non-shortcut version. + +### Resist the temptation of "easy" + +Sure that IDE will help you with a ton of autocomplete stuff and let you +easily build your project, but do you understand what's going on? + +Do you understand how your build system works? If you had to run it without +the IDE, would you know how? + +Can you remember your function names without autocomplete? Isn't there a way +to break/rename things to make them easier to understand? + +Be curious about what goes behind the curtains. + +### Start stupid + +One way to get away from the IDE is to "start stupid": Just get the compiler +and get an editor (ANY editor) with code highlight and do your thing: Code, +build it, run it. + +No, it's not easy. But when you jump into some IDE, you'll think of buttons of +simply "Yeah, it runs that" (which is exactly what IDEs do, by the way.) + +### Logs are for events, not user interface + +For a long time, I used logs to show the user whatever was happening -- +'cause, you know, it's a lot easier to use a single thing instead of two. + +Use the standard output to inform the user of events, standard err to inform +the user about errors but use logs to capture something that you can later +process easily. + +Think about logs of something you'll have to parse to extract some information +at that time, not user interface; it doesn't have to be human-readable. + +### Always use a Version Control System + +"This is my stupid application that I just want to learn something" is not +even a good excuse to not use a version control system. + +If you start using a VCS right from the start, it will be easier to roll back +when you do something stupid. + +### One commit per change + +I've seen people writing commit messages like "Fixes issues #1, #2 and #3". +Unless all those issues are duplicates -- in which two of those should be +already closed -- they should be 3 commits, not one. + +Try to keep a change in a single commit (and by change I don't mean "one file +change"; if a change requires changes in three files, you should commit those +three files together. Think "if I revert this back, what must go away?") + +### "git add -p" is your friend when you overchange + +(Git topic only) Git allows merging a file partially with "-p". This allows +you to pick only the related changes and leave the other behind -- probably +for a new commit. + +### Learn to monitor + +On a previous life, to understand how a system behaved, I added a ton of +metrics: how fast things were going in, how fast things were going out, how +many things were in the middle, how many the job processed... + +It gives a really good view of how a system is behaving. Is the speed going +down? If it is, I can check what is going into the system to understand why. Is +it normal going down at some point? + +Thing is, after this, it is really weird trying to figure out how "healthy" a +system without any monitoring is after that. Checking a system health with just +"Is it answering requests" doesn't fly anymore. + +Adding monitoring early will help you understand how your system behaves. + +### The config file is friend + +Imagine you wrote a function that you have to pass a value for it to start +processing (say, a twitter user account id). But then you have to do that with +two values and you just call the function again with the other value. + +It makes more sense to use a config file and just run the application twice +with two different config files. + +### Command line options are weird, but helpful + +If you move things to config files, you could also help your users by adding +an option to select the config file and expose it. + +There are libraries to handling command line options for every language today, +which will help you into building a good command line and giving your users a +standard interface for everything. + +### Not just function composition, but application composition + +Unix came with the idea of "applications that do one thing and do it well". + +Now, I said you could use one application with two config files, but what if +you need the result of both applications? + +That's when you can write an application that reads the results of the first +one with both config files) and turn into a single result. + +### Even for app composition, start stupid + +Application composition may lead to microservices -- which is good -- but +microservices require some ideas about how applications "talk" between them +over the wire (protocols and such). + +You don't need to start with that. Both applications can write and read from +files, which is way easier. + +Worry about talking over the wire later, when you understand how networks +work. + +## On a Team/Work + +### Code reviews are not for style + +Take your time on code reviews to point architectural or design problems, not +code style problems. Nobody really likes the person whose code reviews are only +"you left blanks in this line" or "missing space before parenthesis" and such. + +Now, if you _do_ find architectural or design problems, _then_ you can add your +code style problems. + +### Code formatting tools are ok, but they are no silver bullet + +One thing a team may be tempted to do to avoid discussing style in code reviews +is to use a code formatting tool to auto-format code before commiting. + +Now yeah, that kinda solves the problem, but there is one small problem: +we, humans, are not as flexibile to read code as computers are; what is +readable by a computer may not be readable by a human. Surely they try to +create some heristics on what is good for human reading, but that doesn't mean +it gets right. + +If you _do_ use a code formatting tool, use it to find out where it changes the +code the most; you probably need to simplify that part of the code to avoid it +messing so much. + +### Code style: Follow it + +If your project have a defined code style, you must follow it. Sometimes it +may not be clear ("this struct/class should be singular or plural"?), but do +your best to follow it. + +### ... unless that code style is the Google Code style + +(Totally personal opinion, feel free to disagree) Every freaking time Google +comes with their own coding style, it's a garbage fire. The community came +with a better style way before and Google seem to come with a style with high +contrasting parts just to call it theirs. + +### There is only one coding style for C/C++: K&R + +(Totally personal opinion again) Every other coding style is _WRONG_. :) + +### There is only one coding style for Python: PEP8 + +The community (most of it) writes code in PEP8. Follow it and your code +smoothly integrate with the rest of the ecosystem. + +### Explicit is better than implicit + +You know what's one of the worst function names ever? `sleep()`. + +Sleep for how long? It is seconds or milliseconds? + +Be explicit with what you use; `sleepForSecs` and `sleepForMs` are not +perfect, but are better than `sleep`. + +(Think about this when you're writing your app command line interface or its +config file.) + +(I could throw the whole "Zen of Python" here, but I'm trying to focus on +personal, direct experience.) + +### Companies look for specialists but keep generalists longer + +If you know a lot about one single language, it may make it easier to get a +job, but in the long run, language usage dies and you'll need to find +something else. Knowing a bit about a lot of other languages helps in the long +run, not to mention that may help you think of better solutions. + +"A language that doesn't affect the way you think about programming, is not +worth knowing." -- Alan Perlis + +For a long time, I kept a simple programming rule: The language I'm playing at +home should not be the same language I'm using at work. This allowed me to +learn new things that later I applied in the work codebase + +### Think of the users + +Think how the data you're collecting from your users will be used -- this is +more prevalent on these days, where "privacy" is a premium. + +If you capture any used data, remember to protect it against unauthorized use. + +### The best secure way to deal with user data is not to capture it + +You can be sure that, at some point, the data will leak, either by some +security flaw or human interference. + +If you don't capture any user data -- or store it in anonymized way -- you +won't have any problems. + +### Keep a record of "stupid errors that took me more than 1 hour to solve" + +I tried but never managed to create a list of stupid errors I kept finding +that took more than 1 hour to solve it, which were simply "forgot to add +dependency" or "add annotation", mostly because there was more than once that +I kept fighting some stupid error for more than 1 hour. + +But you should try to keep a list of stupid errors that took you 1 hour to +solve, 'cause later you can use it to not stay more than 1 hour to solve some +stupid error. + +## Personal + +### When it's time to stop, it's time to stop + +Learn when you can't code anymore. Learn when you can't process things anymore. +Don't push beyond that, it will just make things worse in the future. + +I tried to keep coding once when I had a migraine (not strong, but not mild). +Next day, when I was better, I had to rewrite most of the stuff I did, 'cause +it was all shit. + +### Code of conduct protect _you_, not _them_ + +When you're beginning with any language/library/framework, check their CoC; +they will protect _you_ from being harassed for not immediately getting what +is going on instead of blocking you from telling them what you think. + +I'm mentioning this 'cause a lot of people complain about CoC, but they +forget that they allow them to join in any project without being called +"freaking noob" or "just go read the docs before annoying us". + +Also, remember that most people that are against CoCs are the ones that want +to be able to call names on everyone. + +### Learn to say no + +Sometimes, you'll have to say no: No, I can't do it; no, it can't be made in +this time; no, I don't feel capable of doing this; no, I don't feel +comfortable writing this. + +Once I had to say to our CTO: "Ok, I'll do it, but I want to note that I don't +agree with what we are doing." In the end, the app was barred exactly because +the thing we were doing. + +### You're responsible for the use of your code + +This is hard. Very very hard. It's the difference between "freedom" and +"responsibility". + +There is nothing wrong in writing, for example, a software to capture people's +faces and detect their ethnicity, but you have to think about what that will +be used on. + +### You'll learn about yourself the hard way + +We get frustrated with code that doesn't compile. We get angry with costumers +asking things back and forth. + +And we lash out on other when that happens. + +And that will get you in trouble. + +It happens. + +### People get pissed/annoyed about code/architecture because they care + +You'll find yourself in the other side of the coin: You'll describe some +solution and people will seem annoyed/pissed about some solution. + +When people care about a product/code, they do that. + +"Yeah, you don't like that hushed solution 'cause you care" was one of the +nicest things someone told about myself. + +### Learn from your troubles + +You'll get annoyed, pissed, frustrated, and angry. You'll get you in trouble. +You'll see people getting in trouble because of this kind of stuff. + +You must learn about it. Don't ignore it. + +One thing I learnt the hard way was that I get really aggressive when I'm +frustrated. Now, when I notice I start to get frustrated, I ask help from +someone else. It's really therapeutic to see that someone else also struggles +with your problem, and that's not just you. + +### Pay attention on how people react to you + +I have a "angry man resting face" kind of face. + +Sometimes I'll ask things and people will move a bit back -- like I'm telling +them their solution is wrong. + +That's when I have to add "I'm not saying it's wrong, I'm just confused". + +That _may_ help you to not get in trouble. + +### Learn to recognize toxic people; stay away from them + +You'll find people that, even if they don't small talk you, they will bad +mouth everything else -- even some other people -- openly. + +Stay away from those people. + +You have no idea how that kind of attitude will drive you down. + +### Beware of micro-aggressions + +"Micro-aggressions" are aggressive comments in small doses. Like someone that +keeps calling you "_that_ person" or seemingly innocuous comments about your +position in some policy. + +Those are hard to fight, 'cause PR won't listen to you saying that they are +attacking you. Also, they are hard to detect, 'cause they seem small enough, +but they do pile up and you'll blow your anger all at once. + +Better just stay away and avoid contact as possible. + +### No, I don't think they are "fixable" + +(Personal opinion) Someone could say "Hey, maybe if you spoke to that person, +they would stop". + +Personally, I don't think they would. This kind of stuff is going for so long +to them that it feels natural and, most of the time, you're the wrong one (for +not seeing that they are joking, for example, in true "Schrödinger's asshole" +style.) + +### Toxic/migro-aggressors are only fixable if they are _YOU_ + +Unless it's you realizing you're acting like a toxic person or micro-attacking +someone, and realize that you're actually doing more harm than good being that +way, there is no way to fix those traits (again, personal opinion). + +...mostly 'cause hearing from someone else may feel "_they_ are the ones +against me!" to them. + +### Hero Projects: You'll have to do it someday + +An "hero project" is a project/spec change/framework that you personally think +will solve a group of problems in your project. It could be a different +architecture, a new framework or even a new language. + +That means you'll spent your free time to write something that is already +being worked/exists just to prove a point. + +Sometimes it proves you where wrong. + +(But you got something from it, nonetheless.) + +### Don't confuse "hero project" with "hero syndrome" + +I have seen this at least two times: Someone claims things don't work when +they aren't around or that they don't need help. + +This is "hero syndrome", the idea that that person is the only one capable of +solving all the problems. + +Don't be that person. + +### Paper notes are actually helpful + +I tried to become "paperless" many times. At some point, I did keep the papers +away, but in the very end, it really do help to have a small notebook and a +pen right next to you write that damn URL you need to send the data. + +### Trello is cool and all, but Postits are nicer + +Nothing says "I'm really busy, but organized" like having a bunch of post its +on your desk. + +### Blogging about your stupid solution is still better than being quiet + +You may feel "I'm not start enough to talk about this" or "This must be so +stupid I shouldn't talk about it". + +Create a blog. Post about your stupid solutions. They are still smarter than +someone else's solution. + +Also, come back later and fight your own solutions with better ones. + +Show your growth. + +On top of that, they help you keep small notes or things you need to do. + +### Post your stupid solution online + +Don't keep a Github only for those "cool, almost perfect" projects. You're +free to show that, at some point, you were a beginner. + +You can always come back and improve your code. + +(Or don't: I still have a public repo of my first Python project that looks +like I just translated Java into Python, without the Pythonic part.) + +### Keep a list of "Things I Don't Know" + +Richard Feymann, famous physicist, kept a notebook with the title "Things I +Don't Know". + +When you find something that seems cool and you'd like to know more, create a +file/note/whatever with it in the title. Then make notes about what you +find/figure out.