title: Career advice for junior developers published: true description:

tags: codenewbie, beginners, career, programming

I had a nice chat this morning with our new intern, who's due to start working with us this summer. One of the things he asked me was if I could give him any tips about things he should be looking to learn in order to advance his career in programming. In my mind, I started formulating a list of programming languages, web frameworks, libraries and technologies. I was then struck by the thought that most things on my list would be obsolete in 5-10 years. I considered whether knowing the intricacies of a specific language, or how a framework is put together or how the Model-View-Controller paradigm works, was a solid foundation for a successful career in Software Development. The clear answer to my own question was that no, all these things are transient. They are here today, gone tomorrow. An old proverb sprang to mind: "You give someone a fish, they'll eat for a day. Teach them to fish and they'll be eating for a lifetime".

So this is what I told him: "Learn whatever catches your eye, excites you and you feel comfortable with. But whatever you do, keep in mind the following":

1. Enjoy what you do

You'll be spending one of the biggest parts of your life working (the other big part is sleeping but we can't help that). If the tools or people you're working with frustrate, impede or depress you, then change them. Sometimes, this may require changing jobs. That's fine, you have no obligation to suffer in exchange for money or out of some mis-placed sense of loyalty. By enjoying what you do at work, you're going to be better at it and more productive. Enjoying being round your colleagues will make you more collaborative and help you learn new things. Most importantly, by working in an enjoyable environment you will also be safeguarding your mental health, this most precious of goods.

2. Coding isn't enough

To be a good developer you need to be able to code. To be a great developer you need to be able to:

• Communicate your assumptions and expectations to others.
• Empathise with others and understand their point of view.
• Plan ahead with forethought and discretion.
• Manage other people's expectations of you.

Unfortunately, there aren't any boot camps for these skills. These are skills you need to hone and cultivate yourself. They take time and effort and not everyone manages to accomplish all of them. But mastering these skills can make all the difference between yet another coder and an effective software developer.

3. You read more code than you write

You may not think so, but, trust me, you do. So what does this mean? It means that when you write code you must keep in mind that someone else (or even worse, you 12 months later) will have to read it. So make sure your code is readable, well-documented and trimmed down to what is needed to deliver the intended piece of functionality. Ensure your codebase is not bloated with clever code hacks or complex design patterns and architectures which only exist to inflate people's egos or adhere to some dogmatic pattern or methodology. Keep your code simple, modular and readable. Your colleagues and future self will thank you for it.

4. Separate the things that change from the things that stay the same

Everything changes over time, but there are things that change at predictable occasions or at regular frequency. Make sure these things are separated from stuff that doesn't often change. So keep your UI code well away from your business-logic code. Keep your data-reading code away from your data-formatting code. If you are dealing with external APIs, hide them behind an adaptor or facade. The APIs will be changing a lot more frequently than your internal usage of them, i.e. your facade. Apply this principle throughout your code and designs.

5. Develop iteratively and incrementally

Never try to build a complete and perfect system in one go. Focus on the most needed and clear features first and provide the simplest working solution to them. Then build-up your project over many iterations, each iteration refining existing features and delivering new ones.

If you're not sure what an iterative and incremental approach is, then this should help clarify it.

6. Premature optimisation kills projects

Do not try to anticipate future needs without solid evidence that these needs are likely to arise. If your web-site ever needs to handle 10 zillion requests per minute then don't worry, at that stage you'll be making enough money to comfortably retire at your own private island, so focus on that instead ;) .The point is, do not optimise pre-maturely, 'just in case'. Do not work on the basis of 'what-ifs'. If you do, you'll only add unnecessary complexity in your code, increase the likelihood of bugs and reduce its readability. On that note, prefer languages and frameworks that promote flexibility and ease-of-change.

7. No Silver Bullet

To quote Mr Brooks

There is no single development, in either technology or management technique, which by itself promises even one order of magnitude [tenfold] improvement within a decade in productivity, in reliability, in simplicity.

Over your career you'll see many 'hotnesses' come and go. You'll hear people raving on about the fastest web framework, the coolest design paradigm, the best programming language, and so on. All these things may be true within the narrow slice of reality of the people clamouring for them, but they are not necessarily true in your reality. Complex problems, like the ones we face in software development, are never solved by simple solutions, like using a single programming language or web framework. Never be fooled by thinking you found that one thing which will solve all your programming problems.

8. Generalise, don't specialise

In nature, the species of animals more likely to survive are the generalist species, that is the species which can exploit different environments and food sources. In contrast, the species more likely to go extinct are the specialists, the ones that rely on a single food source or can only thrive in one environment. The same applies in software development. Do not put all your eggs in one basket, as the saying goes. The ability to do many things well, will see you farther than the ability to one thing brilliantly.

Epilogue

Here you go, I'm sure there are a couple of more items I forgot to add, but I think these capture the fundamental principles I've learned over many years as a software developer. I wish someone told me these things when I was first starting out. Hopefully they may help someone else, so thanks for reading and feel free to add your own advice in the comments.

Our assignment

Our company has just gone on a huge recruitment spree and has just hired 100,000 new employees. The HR department has sent us a spreadsheet with the employee details, asking us to save them in the company staff database. After having a look at the data, we notice a few actions we need to take before inserting the details into our database and we come up with the following algorithm:

• Read data from the spreadsheet

• Break employees' full name into first name and last name fields

• Assign each employee a unique company number

• Insert employee into the database as long as their first and last name is less than 255 characters long

A single-process ETL pipeline

We then proceed to write a crude but effective ETL pipeline

require 'csv'
require 'pg'
require 'ostruct'
require 'securerandom'

@users = []
@conn = PG.connect( dbname: 'testdb' )
DATA_FILE = "data/users.csv"

def extract(data_file)
  CSV.foreach(data_file) do |row|
    user = OpenStruct.new
    user.name = row[0]
    user.email = row[1]
    @users << user
  end
end

def transform
  @users.map! do |usr|
    usr.first_name = usr.name.split[0]
    usr.last_name = usr.name.split[1]
    usr.staff_no = SecureRandom.uuid
    usr
  end
end

def load(usr)
  if( usr.first_name.length < 255 &&  usr.last_name.length < 255)
    @conn.exec_params(
      "INSERT INTO STAFF (FIRST_NAME, LAST_NAME, EMAIL, STAFF_NO) VALUES ($1, $2, $3, $4)",
      [usr.first_name, usr.last_name, usr.email, usr.staff_no]
    )
  else puts "Validation failed for #{usr.first_name} #{usr.last_name}"
  end
end

extract(DATA_FILE)
transform
@users.each {|usr| load(usr)}

Our extract method reads the data from file and stores it in memory. The transform method then splits the full name field into first and last name and assigns a UUID to the employee. Finally, our load method ensures the name fields don't exceed the expected length, before inserting the data into the Staff table.

We test our script and it works fine, however all this data crunching takes a long time and, us being the perfectionist that we are, would like to make it faster.

The need for speed

Enter Jongleur a Ruby gem that allows us to create tasks that are executed as separate OS processes and that can be ran with a certain precedence.

We'll use Jongleur to split our ETL pipeline into two separate and parallel pipelines, each dealing with one half of the input data. We want our task graph to look like this:

Each Extract, Transform and Load task will do exactly what we described in our initial algorithm above, but with only half of the spreadsheet records. The Cleanup task will kick-in last, to sweep away any undue remnants after our pipeline has been ran.

Representing the above graph in Jongleur is a simple matter of creating a Ruby Hash:

etl_graph = {
  Extract1: [:Transform1],
  Transform1: [:Load1],
  Extract2: [:Transform2],
  Transform2: [:Load2],
  Load1: [:Cleanup],
  Load2: [:Cleanup]
}

We can then just tell Jongleur to use our task graph

Jongleur::API.add_task_graph etl_graph

Design considerations

One thing we need to consider before creating our parallel tasks is how to share data between tasks which are ran as separate processes. Shared Memory could be used but has some pitfalls which is why we'll be using the wonderful in-memory data store which is Redis. Redis is (mostly) single-threaded, which means concurrent requests are executed sequentially and safely (and blazingly quickly). Furthermore, we'll take advantage of one of Jongleur's great features whereby each task can access the process ids (pids) of its predecessors. This means that we can use a task's pid as a key for its data. So our Extract1 task can save data to Redis with its own pid and when Transform1 task starts it will know exactly which data was created by Extract1, its predecessor!

Common attributes

Every Jongleur task class must inherit from the base class Jongleur::WorkerTask. This means that we can use this class to define data that we want accessible from any and all WorkerTask instances. In our case we want any Extract, Transform and Cleanup tasks to be able to access our Redis data store. To achieve that we simply define the Redis connection as a class variable in the Jongleur::WorkerTask hierarchy:

class Jongleur::WorkerTask
  @@redis = Redis.new(host: "localhost", port: 6379, db: 15)
end

Extracting

Both our Extract1 and Extract2 tasks will need to have some common functionality, namely to parse records from the csv file and save them to Redis:

class Extract < Jongleur::WorkerTask

  DATA_FILE = "data/users.csv"

  def process_row(row, rowno)
    user = {}
    user[:num] = rowno
    user[:name] = row[0]
    user[:email] = row[1]
    user

    @@redis.hset(Process.pid.to_s, "user:#{rowno}", user.to_json)
  end
end

Now the only divergence between Extract1 and Extract2 comes when reading the input data from the csv file. Extract1 will only read and store the first 50,000 records, while Extract2 will read and store the remaining 50,000 records.

class Extract1 < Extract
  def execute
    CSV.foreach(DATA_FILE).with_index(1) do |row, rowno|
      break if rowno >= 50000
      process_row(row, rowno)
    end
  end
end

class Extract2 < Extract
  def execute
    CSV.foreach(DATA_FILE).with_index(1) do |row, rowno|
      next if rowno < 50000
      process_row(row, rowno)
    end
  end
end

Transforming

Each Tranform task will need to use its Extracting predecessor's pid in order to load the extracted records it needs to transform. It will then proceed to split each record's name, assign it a UUID and save it back to Redis:

class Transform < Jongleur::WorkerTask
  @desc = 'Transforming'
  def execute
    loader_id = Jongleur::Implementation.get_process_id(predecessors.first)
    users = @@redis.hgetall(loader_id)
    users.keys.each do |usr_key|
      transformed_user = {}
      data = JSON.parse users[usr_key]
      transformed_user[:first_name] = data['name'].split[0]
      transformed_user[:last_name] = data['name'].split[1]
      transformed_user[:staff_no] = SecureRandom.uuid
      transformed_user[:num] = data['num']
      transformed_user[:email] = data['email']
      @@redis.hset(Process.pid.to_s, "user:#{data['num']}", transformed_user.to_json)
    end
  end
end

class Transform1 < Transform; end
class Transform2 < Transform; end

Loading

Load1 and Load2 will pick up the records stored by their respective predecessor Transform tasks, will validate the name length and will insert the records in the Staff table of our HR's Postgres database:

class Load < Jongleur::WorkerTask
  @desc = 'Loading'

  def execute
    loader_id = Jongleur::Implementation.get_process_id(predecessors.first)
    users = @@redis.hgetall(loader_id)
    conn = PG.connect( dbname: 'testdb' )
    users.keys.each do |usr_key|
      data = JSON.parse users[usr_key]
      if( data['first_name'].length < 255 &&  data['last_name'].length < 255 )
        conn.exec_params(
          "INSERT INTO STAFF (FIRST_NAME, LAST_NAME, EMAIL, STAFF_NO) VALUES ($1, $2, $3, $4)",
          [ data['first_name'], data['last_name'], data['email'], data['staff_no'] ]
        )
      else puts "Validation failed for #{usr.first_name} #{usr.last_name}"
      end
    end

  end
end

class Load1 < Load; end
class Load2 < Load; end

Cleanup

All our Cleanup task has to do is to erase the Redis data created by the previously-ran tasks

class Cleanup < Jongleur::WorkerTask
  def execute
    @@redis.flushdb
    puts "...Cleanup"
  end
end

Running our pipelines

We now have

1. Defined our Task Graph and configured Jongleur with it

2. Implemented an #execute method for each of the Tasks in our Task Graph

We are now ready to invoke Jongleur and run our parallel pipelines:

API.run do |on|
  on.completed do |task_matrix|
    puts "Jongleur run is complete \n"
    puts task_matrix
    puts "oh-oh" if API.failed_tasks(task_matrix).length > 0
  end
end

The output is:

Starting workflow...
starting task Extract1
starting task Extract2
finished task: Extract1, process: 22722, exit_status: 0, success: true
starting task Transform1
finished task: Extract2, process: 22723, exit_status: 0, success: true
starting task Transform2
finished task: Transform1, process: 22726, exit_status: 0, success: true
starting task Load1
finished task: Transform2, process: 22727, exit_status: 0, success: true
starting task Load2
finished task: Load1, process: 22729, exit_status: 0, success: true
finished task: Load2, process: 22732, exit_status: 0, success: true
starting task Cleanup
...Cleanup
finished task: Cleanup, process: 22745, exit_status: 0, success: true
Workflow finished
Jongleur run is complete
#<struct Jongleur::Task name=:Extract1, pid=22722, running=false, exit_status=0, finish_time=1546800512.6342049, success_status=true>
#<struct Jongleur::Task name=:Transform1, pid=22726, running=false, exit_status=0, finish_time=1546800518.355283, success_status=true>
#<struct Jongleur::Task name=:Extract2, pid=22723, running=false, exit_status=0, finish_time=1546800512.967079, success_status=true>
#<struct Jongleur::Task name=:Transform2, pid=22727, running=false, exit_status=0, finish_time=1546800519.273847, success_status=true>
#<struct Jongleur::Task name=:Load1, pid=22729, running=false, exit_status=0, finish_time=1546800533.147315, success_status=true>
#<struct Jongleur::Task name=:Load2, pid=22732, running=false, exit_status=0, finish_time=1546800534.1630201, success_status=true>
#<struct Jongleur::Task name=:Cleanup, pid=22745, running=false, exit_status=0, finish_time=1546800534.899818, success_status=true>

Jongleur gives us a thorough account of each Task's details and success status. Notice how both initial Tasks (Extract1, Extract2) start at the same time and each new Task starts only once its predecessor on the Task Graph has finished. If a Task failed, for whatever reason, Jongleur wouldn't execute its dependent tasks but it would continue running any other tasks it could. The status of all Tasks would still be visible in the Task Matrix structure yielded by the completed callback.

Performance

Running the simple single-process ETL pipeline at the beginning of this article produced the following timings:

9.55s user 1.40s system 32% cpu 33.494 total

Running the parallel Jongleur pipelines produced:

19.99s user 5.23s system 89% cpu 27.210 total

User time is increased with Jongleur (I'd guess due to using Redis) and so is kernel time, obviously due to forking and managing many processes, but overall time is approx. 18% faster using Jongleur with two pipelines. Also consider that in this example we only used two parallel lines, i.e. we split the data load into 2 parts. It is a simple matter to split it into 4, 8 or even 16 parallel lines (dependent on our CPU cores) and thus gain a massive performance gain, using the same logic and principles demonstrated here and simply adding more Tasks.

Jongleur advantages

1. Performance, through parallelisation.

2. Implicit precedence handling. Jongleur will start running a task only once its predecessor tasks have successfully finished running. If a task fails, its dependent tasks wil not be ran.

3. Code modularisation. If a task fails, for whatever reason, it will be marked as such by Jongleur and its dependent tasks will not be run. However, that won't stop any other tasks from being run, which means that partial failure does not necessitate complete failure, in use cases where this is desirable.

The Complete Code

including data file for this demo is freely available

The Future

Feel free to suggest improvements and new features for Jongleur or to talk to me about trying it out for your own Use Cases :)

FizzBuzz is a simple kids game, often used as a test at programming interviews. It goes like this:

"Write a program that prints the numbers from 1 to 100. But for multiples of 3 print “Fizz” instead of the number and for multiples of 5 print “Buzz”. For numbers which are multiples of both 3 and 5 print “FizzBuzz”.

I fell asleep yesterday while thinking about all the different ways we can solve FizzBuzz in Ruby. Then three ghosts appeared in my dream. The first one was the Ghost of Imperative Programming (GImP, for short). It showed me how to do FizzBuzz in an imperative way:

def imperative(n)
  if (n % 3 == 0) && (n % 5 != 0)
    'Fizz'
  elsif (n % 3 != 0) && (n % 5 == 0)
    'Buzz'
  elsif (n % 3 == 0) && (n % 5 == 0)
    'FizzBuzz'
  else n.to_s
  end
end

"Simple conditionals are easy to read", it said. "They're also relatively fast. They're a good, basic way to implement your solution"

Next, the Ghost of Antecedent Declarative Inference appeared (GhAnDI). "Don't listen to GImP" it said. "Be more clever. Look here, the conditions that are evaluated to produce each FizzBuzz value are mutually exclusive, if you represent them as Hash values then only a maximum of one of our Hash keys will have the value of 'true' for any given number. Use this to your advantage."

def declarative(n)
  h ||= {
    'Fizz' => (n % 3 == 0) && (n % 5 != 0),
    'Buzz' => (n % 3 != 0) && (n % 5 == 0),
    'FizzBuzz' => (n % 3 == 0) && (n % 5 == 0),
    n.to_s => (n % 3 != 0) && (n % 5 != 0)
  }
  h.key(true) || n
end

"You then make sure you return this 'true' key (h.key(true)) from the method. Of course if all the keys have the 'false' value (i.e. the passed number is neither divisible by 3, nor by 5) then h.key(true) will evaluate to nil, so use the logical OR operator to ensure that -in this case- you return the actual passed number."

"That's pretty smart" I said. "We are not specifying a sequence of steps leading to the solution, but rather the conditions required and we leverage the language features to work out the correct one. Cool!"

As GhAnDI departed, I sensed a new presence arriving. It was wearing trendy clothes , had a well-oiled beard and there was a large and vocal group of developers following it around. It was... the Ghost of Functional Erudition (GoFEr).

"The other two are the past", GoFEr said. "I will show you the future"

def functional(n)
  [[15, 'FizzBuzz'], [5, 'Buzz'], [3, 'Fizz']].select { |x| (n % x[0] == 0)}.map{|a| a[1] }.first || n.to_s
end

"We specify both conditions and outcome together" it said. "Ruby doesn't have tuples, but we can get a similar effect with an Array of Arrays. We then select the 'tuple' which satisfies the condition of zero-mod division with the tuple's divisor (x[0]). If more than one tuples match, we only select the first one, since we cleverly ordered our Array that way. And if there are no matches then we return the original number input. Simples."

And with that GoFEr shimmied away to the sounds of 'California dreaming' and I woke up. I learned some useful techniques from this encounter, but mostly I came to appreciate just how powerful and flexible the Ruby language really is.

Merry Christmas everyone and happy Ruby-ing!

PS: As a Christmas present, the Ghosts put all their code along with some bench-marking on GitLab (https://gitlab.com/snippets/35596)

Intro

The Proxy is one of the most popular design patterns in most programming languages. A Proxy is simply an object that sits between some client code and a service object. The client code deals directly with the Proxy, instead of the service object. ¹ This means that the Proxy can be used to mask the service object's physical location (a Remote proxy), manage access to the service object (a Security proxy), or just lazily initialize the service object (a Virtual proxy).

The old way

Whatever the use-case, the idiomatic Ruby way to implement a proxy has been by utilizing the method_missing method, as described in Russ Olsen's seminal book Design Patterns in Ruby.

# the service object
class Account
  def initialize(balance)
    @balance = balance
  end

  def deposit(amount)
    @balance += amount
  end

  def withdraw(amount)
    @balance -= amount
  end

  def self.interest_rate_for(a_balance)
    a_balance > 10_000 ? '3.2%' : '5.5%'
  end

end

# the proxy object
class AccountProxy
  require 'etc'
  def initialize(real_account)
    @real_account = real_account
  end

  def method_missing(name, *args)
    raise "Unauthorised access!" unless Etc.getlogin == 'fred'
    @real_account.send(name, *args)
  end
end

## client code
acc = Account.new(100)
proxy = AccountProxy.new(acc)
proxy.deposit(10)
proxy.withdraw(50)

In this simple proxy example, we intercept all method calls to AccountProxy, do a basic security check and then delegate the method to the real Account object to do the actual work.

This is a concise and effective way to create a proxy in Ruby, but it has some weaknesses:

1. The client code needs to call a separate Object, e.g. AccountProxy instead of Account. It's pretty obvious we're not dealing with the 'real' service object, which is not only a nuisance if you have to re-write existing client code, but may also lead malicious actors to try to by pass the proxy.

2. method_missing is a catch-all trap. It will catch every method call coming our way and we need to think long and hard about which methods we want to delegate to the 'real' object and which ones to handle in our proxy (#to_s, for instance)

3. Using method_missing has a performance hit. When calling a method on an object, the Ruby interpreter will first look all the way up the object hierarchy trying to find the method and -when it can't- will go back down the hierarchy tree and start looking for a method_missing implementation. This will happen for every single method call.

4. Our proxy doesn't work with class methods. To do that we'd need a separate proxy object for Account's singleton class.

The new way

A few years after Russ's book came out, Ruby 2.0.0 was released and introduced Module#prepend. The way Module#prepend works is by inserting the prepended Module between the calling code (a.k.a the 'receiver' object) and the module or class that does the prepending.

Can you see the connection already? The prepended Module sits between the receiver and a service object. Sounds familiar? Oh yes, this is exactly what a Proxy is meant to be doing!

Knowing this we can use the #prepended hook method to dynamically implement all methods of the prepending module in our proxy, making sure to add our extra security check, before calling the original method implementation.

module Proxy
  require 'etc'

  # Dynamically re-creates receiver's class methods, intercepts calls 
  # to them and checks user before invoking parent code
  #
  # @param receiver the class or module which has prepended this module
  def self.prepended(receiver)
    obj_mthds = receiver.instance_methods - receiver.superclass.instance_methods
    obj_mthds.each do |m|
      args = receiver.instance_method(m).parameters # => [[:req, :x], [:req, :y]]
      args.map! {|x| x[1].to_s}
      Proxy.class_eval do
        define_method(m) do |*args|
          puts "*** intercepting method: #{m}, args: #{args}"
          raise "Unauthorised access!" unless Etc.getlogin == 'fred'
          super(*args)
        end
      end
    end
  end

end #module


class Account
  def initialize(balance)
    @balance = balance
  end

  def deposit(amount)
    @balance += amount
  end

  def withdraw(amount)
    @balance -= amount
  end

  def self.interest_rate_for(a_balance)
    a_balance > 10_000 ? '3.2%' : '5.5%'
  end

  prepend Proxy

end

All we're doing is taking advantage of the way #prepend affects the Ruby Object Model in order to find out which methods the intercepted object is defining and then implementing them in our proxy while adding our own code. To call the original implementation, we once again leverage the fact that the intercepted object is the parent of our proxy module, so all we need to do is call #super (no more ugly #send calls)

Let's write some client code to exercise our account:

acc = Account.new(100)

puts acc.deposit(10)
puts acc.withdraw(50)
puts Account.interest_rate_for(2000)

which outputs:

*** intercepting method: deposit, args: [10]
110
*** intercepting method: withdraw, args: [50]
60
5.5%

That's pretty cool, but the beauty of it doesn't end here. We can use the very same mechanism to intercept class method calls. All we need to do is prepend the Proxy to the Account's singleton class:

class Account
  def initialize(balance)
    @balance = balance
  end

  def deposit(amount)
    @balance += amount
  end

  def withdraw(amount)
    @balance -= amount
  end

  def self.interest_rate_for(a_balance)
    a_balance > 10_000 ? '3.2%' : '5.5%'
  end

  class << self
    prepend Proxy
  end

  prepend Proxy

end

Once again, the Object Model is manipulated to our favour:

And now by running our client code again, we see that both instance and class methods of the Account class are caught by the proxy:

acc = Account.new(100)

puts acc.deposit(10)
puts acc.withdraw(50)
puts Account.interest_rate_for(2000)

which outputs:

*** intercepting method: deposit, args: [10]
110
*** intercepting method: withdraw, args: [50]
60
*** intercepting method: interest_rate_for, args: [2000]
5.5%

This way of writing proxies has a number of advantages:

1. It's transparent. Client code doesn't need to change, it doesn't even need to know that a proxy is being used, we just need to prepend the proxy module to the service object class.

2. It's performant. Some basic bench-marking has indicated a 3-4x performance gain against a method_missing proxy.

3. It works for both instance and class methods.

So there you have it: Another way of building proxies. Ruby keeps evolving and so do we. Happy 25th birthday!

1: a Ruby object that encapsulates some business logic