How do you handle being the only tech guy in a non tech team?

Question

The company I currently work for has a terrible amount of processes that require manual work, thus they are very limited in the amount of projects they can take, as their processes do not scale up, they have reached a point where they have to reject projects. These processes largely depend on manual setups, manual execution and writing Excel and Word documents and the like, an example of this, one of these processes involves filling more than 800 cells in a spreadsheet manually, another one exporting all those cells manually to another internal tool and so on.

I was hired by this company around six months ago with a technical lead role, with the intention of transitioning the current department I am in from manual work to automated work with Python as a programming language. My profile is purely technical, I am a software developer and software architect with a decade of experience.

During these six months, I have performed two different roles, building the Python infrastructure to automatically perform the processes and also use this same infrastructure to deliver results in a very important project they have, this is all working fine and I have been praised by the high quality of the deliverables.

The infrastructure is as generic and flexible as it can be, and it can be customized via Python to satisfy the different needs of the projects. The logic is the same as for example project in React or in Django. Everything complicated is abstracted out in the infrastructure and available via a function call in Python, it is all designed so that the user only needs to program project specifics and combine blocks together using Python.

The problem is, because other people in the department have been busy with other projects and the manual processes take so long, no one other than me has been working in this infrastructure or the projects derived from it, and no one else in the department is a programmer nor knows anything about Python, the little programming they have done is VBA.

I showed my colleagues how everything worked and they recognize the value it delivers and find it amazing, but completely dislike Python, implementing the project specifics or maintaining the infrastructure. They are not programmers and the role they are supposed to transition to requires programming to a certain degree, implementing the project specifics requires knowledge of simple Python so I was hoping they would start from there in each project and witness the results themselves, but I am starting to believe this is never going to happen with the excuse they are busy doing the manual work in other projects.

I have recently learned that an external company was hired in the past for the same thing I am doing but didn't get far.

The other problem is that neither my manager nor other higher ups know anything about software, there are no hours allocated for the infrastructure so I put all the hours I spent doing the infrastructure, which is not project specific, to a specific project, as I was told, and now they believe that automated processes take too much time because that project took longer than expected, despite me explaining multiple times that most hours are infrastructure related and very little in actual project functionality.

So I am looking for advice as I feel alone and I am not sure if I am hitting a wall, right now I have limited the growth of the infrastructure before it becomes too much for one person to handle, and focusing on gathering interest and getting proper statistics of what is possible to do and how much time it takes.

EDIT:

I will provide more context based on the answers provided.

A process itself can consist in measuring a physical magnitude 800 times and reporting the results between certain thresholds or some criteria in an Excel spreadsheet 800 hundred times, right now this is done manually by a well paid engineer, who presses the button of a tool 800 hundred times every five seconds or so (as fast as he can) and does the filtering in his head and writes the data to a spreadsheet manually.

The communication with the tool can't be done with VBA for several reasons including technical limitations of the language, nor this would scale or be maintainable to all the tools that measure that magnitude because they use different protocols and belong to different manufacturers.

Note this is just a process in a huge framework of processes and projects that vary a lot and can change, and note that this is also required to be in real time despite it not being done in real time due to the existing technical limitations of it being done manually, customers have complained but it is not addressed. This is the reason why they praised the results of my work, because this kind of concerns were also addressed.

So I need a proper language, and I chose Python because I know it well and it is the easiest to learn and start with, with Python I can handle all this well enough in real time and abstract away all the technicalities of tools, drivers and so on and just I expect from the user some Python code to establish the filtering criteria for the results and where the tool is located by its IP. I personally do not find this complicated.

I could abstract this to a graphical user interface to avoid users using Python, but note that if I do this for every process, it would mean I would be implementing everything piece of functionality myself in every project while my colleagues just press a button. And not all processes adapt well to a GUI, they require some logic to be implemented via Python, I would not be able to handle this volume of work myself nor it is the intention, the intention is that people start automating things like I am doing myself. The intention is that I handle the infrastructure and complex problems myself, while they automate the needs for other projects with this infrastructure using Python.

I spoke with everyone involved and shared the vision for the infrastructure before starting and explicitly mentioned that Python would be used unless and the multiple reasons for this, specifically the growing complexity of the projects and the fact that the current way of working implied that a lot of customer requierements were not being addressed. I now realize that perhaps they said OK without thinking too much.

The adoption of this tool is being encouraged by giving me time to show other members how it works, and two people have been selected to start working on it with me, and I have to teach them as much as I can throughout this year, but I am not sure if they are happy about it, and if you do not like programming life can be hell.

By generic and flexible as it can be I do not mean it is complicated or there is too many pieces of functionality, it means that they have a single entry point to a functionality regardless of the setup, and the infrastructure handles all the underlying details of communication, protocols and algorithms behind the scenes, they can use whatever tool is available with the infrastructure and reproduce the same results, by generic and flexible I mean that it is not dependant on specific tools or hardware, you simply call a function in Python to get a value from a combination of tools and it handles all the details and intercommunication for you and gives you a value.

In my oppinion, I find that learning a bit of Python is required for what the company intends to do with the department. The company is explicitly getting projects that require a high level of custom automation of processes to get results in very short timeframes.

REGARDING NON-TECH

I am supposed to be the technical lead for the transition, but the title is the same as everyone else in the department if not for the senior tag. By non tech I did not mean that I am working with people from other areas, I meant that I do not consider the manual work that is being done in the department as technical as anyone can press buttons and fill spreadsheets. By other companies standards, we are way behind in terms of processes and timeframes.

Answer 1

So you're saying that the staff at your company are non-technical and have bodged together some spreadsheets and VBA so that they can do their jobs, and that you've replaced this with a system that requires them to learn Python?

The system you've built may be more extensible, maintainable, scalable etc. etc. but if the staff need to write code in Python in order to set up projects and maintain the infrastructure then you're in a losing position, it's not going to happen. Indeed, it shouldn't happen, presumably the staff have other areas of business knowledge that are central to their jobs; customer support, account management, business analysis? Coding is a waste of their time and expertise.

It's entirely understandable that you're in this position. You had to get the bones of the system writen first. You need to protect the data, a UI can seem like a luxury at this stage. Especially when everything hangs off your softare and you're the only person doing the work. However you need to make the system work without relying on the coding ability of the users.

I imagine you're under a great deal of pressure over this. It appears that, alone, you've already done better than a whole company of external consultants. Now is the time to leverage that success to get help. Can you persuade the company to hire someone to give you a hand? If not to help you write a UI then at least to take the load off the users, freeing up your time to start on a UI. You don't need to hire a senior dev at this point, an eager newbie that you can train up may be a better option.

One final thing: I've been in the position of replacing monster Excel spreadsheets containing nightmare VBA macros with properly coded systems and an approach that worked was to keep Excel as the UI in the first iteration to avoid the 'not invented here' problem. Over time they saw that we were taking away their pain and making their lives easier, and they came to trust us when it came to finally dumping Excel.

Answer 2

This sounds like an uncommon (and frustrating) situation, which I haven't personally been in, so likely someone else will have more directly relevant experience. However, this part threw up a red flag for me:

The infrastructure is as generic and flexible as it can be...

This at least sounds like what I've seen some brand-new colleagues do in software jobs. It jumps out out me as possibly being too complicated for an initial piece of work, before any value has been demonstrated. The problems you describe in later paragraphs are pretty much spot-on to my worries when I read that.

I would think that you need to bite off very small pieces of functionality to start with, and prove to yourself and your workplace that you can actually deliver efficiency improvements. Have some conversations with people on your specific project about (a) what the most serious pain-points are, and (b) brainstorm what kind of tool or interface they'd be okay to add to their process. If your solution requires user programming, and they're not programmers, then that's a non-starter. (If they're used to looking at Excel, then maybe making a spreadsheet template with some Python macros under the hood is a reasonable starting alternative.)

Get some buy-in or interest for a small tool in advance. Get it to them in a week. Get some feedback and iterate. Get some velocity that you reduce pain for other people. Show that the project-specific time is actually generating payoffs. You almost have to take the mindset of a tiny startup/skunkworks, finding ways to sell yourself to make the customer happy. Doing a big moonshot and trying to solve all the problems at once (without advance buy-in on the interface) is unlikely to be productive.

In my field (video games) I have seen a single developer independently attached to a non-tech department (art), who was very aware and sensitive to artist needs (e.g., peak facility with visual-based graphical thinking) who was very successful and came to be considered an invaluable team member. She could rapidly take calls for help from the artists and improve the previous text-based tools to ease the bottlenecks there.

Maybe spend a weekend thinking about what your one-man software development process is like. Even the traditional SDLC waterfall model starts off with copious customer interviews in the initial steps. Don't think you can skip that.

Answer 3

It was a major mistake for your to associate infrastructure hours to a particular project. From the onset, you should have ensured that there was a dedicated project for this.

I also suspect it was a mistake for you to go off an implement a system that doesn't appear to be usable by anyone else. Anyway, what's done is done.

I have a feeling this particular issue isn't THAT uncommon.

You need to find some hook so people can start to use the underlying infrastructure, and get some initial productivity gains which frees them up to learn more. You should probably do this in such a way that allows them to use the tools they are familar with to start off with.

Just as an example, if they are used to modifying data via scripts in an excel worksheet, you should consider providing a odata endpoint which can be read/written by excel. You could make this endpoint configurable to perform some data transformations according to configuration files. Then it becomes a matter of slowly, over time, migrating the common transformations from the VBA scripts into features of your endpoint.

Power users may eventually realise they can write their own odata endpoints, or may even decide to jettison excel completely in order to hit the infrastructure directly.

I suspect you need to find the project that is most likely to benefit from this with the least amount of effort and target that.

Answer 4

You have hit a wall. Unfortunately, to get more done, you may need to change - either company or what you are doing.

You built a good system, but nobody can use it. This is a typical mismatch between a technical person and the users.

One solution would be to change the approach. The "customers" for your work are your co-workers. In my experience, it is best to figure out what type of solution my customers can use before I start building something. In this case, it sounds like the customers can use solutions that are Excel and VBA based but not anything else. Yes, VBA is not as clean as Python, but it can still get the job done.

Is there a way to take what you have already written and rewrite it in VBA? I find that giving people a short VBA macro that gives them a hand makes a big difference in their acceptance of what I am doing. Repeat.

Answer 5

You are in a difficult situation, basically being parachuted into a company to automate their old manual processes, with little or no support from senior management, little or no support from your co-workers, and the knowledge that a previous attempt to do this job ended in failure.

Your objective of getting your co-workers to learn Python is good, but not in the short term; quite understandably, they'll ask 'what is in it for me', and a response of 'maybe-somewhat-reduced workload in 6 months' won't seem satisfactory.

Also don't under-estimate the fear-factor of any changes to the status quo; your co-workers may be worried that their non-existent coding skills won't be up to the task, and management will fire them because they can't code. This fear may be completely unfounded, but it can subliminally colour their judgement when it comes to getting involved.

So you need to create a flagship project, that demonstrates how wonderful and non-threatening your approach is. Pick a particularly tedious manual task, that is generally disliked by everyone, and automate it. In the short term you'll have to do all the coding, but when the users ask for improvements, you could gently suggest they change the code, under your watchful eye. If you manage to improve one person's workload, word will get around, and gradually others will show interest - but sadly this will take quite a while, and it wouldn't be surprising if you ran out of enthusiasm for a job that feels like wading through treacle, and sought out a company that was more receptive to your talents.