DevOps MUST Build Internal Developer Platform (IDP)

Hello Viktor, hi channel. I'm happy to have found your channel. I really don't want to sound like a silly fanboy, but what you laid out in this video makes so much sense. I just started with building my own development platform or system. I did not exactly know, what I need and how to call this, but IDP is pretty much it. It is all about abstracting the complexity away from the customers. In our case these customers are our devs, as you mentioned. My primary obstacle at this point is the fact, that I need to learn how to leverage the Kubernetes API as a cloud native tool for an on premise oriented development and deployment toolkit. A very interesting aspect is the extensibility of the Kubernetes API. I really need to figure out, how to use it and how to build CRD's for tasks like bare metal installations or setting up virtual servers on e.g a single VPS server or on a local machine. No idea, if this is possible, but worth a try imho. Building an IDP with very limited local resources could well turn out to be a mad mans plan, but at least worth trying imho. You mentioned not to build it from scratch but to use and adapt existing tools. Just started to look around and I guess in later videos from you I will find more info. I'm glad you are around. I will sure have to consult this channel quite a lot in the coming months. Thanks, Viktor.

Hvala Viktore , odlican video , cist i jasan! Odlicna produkcija! Pozdrav!

Your content is pure gold, trully thank you !

Damn! very informative thanks a lot! we are about to deploy one of our apllication on EKS in order to slowly move another 2 or 3 of our apps. Would you recommend to build the IDP from the very beginning or is it overkill to have an IDP for 3 apps? I am still uncertain if IDP is made for intensive production need or if it's also recommended for small needs. I guess it takes a lot of time to put an IDP together. thanks a lot again for the gold mine!

@Viktor ...u changed the whole perspective man !!! Kudos to ur way of knowledge transfer !!!!

stop talk too much all of you and give at least one example pf such a platform somewhere , i still didn't see any and to say of course they are all inside corporations is some bullshit

I would go with Backstage. Fully flexible

Argo Workflows is absolutely fantastic! I just wish their community had better support though...

Thank you for giving this overview! Briliant! Career changing even. I will be looking for companies who think like that

you described my life, and predicted my future.

Awesome video Viktor. I watched this after "DevOps is Dead", the two videos dovetail into each other perfectly. Thank you!

Thank you always for your videos. Are you planning to have an IDP series where you show how to actually build this IDP?

I completely agree with every word. In fact, I've built my own IDP without knowing this acronym. It is called Kubero and is open-source and completely free to use.
It has a UI, API, and CLI. It is entirely kubernetes native. It also follows the workflows from Heroku. So it is very easy for DevOps to start a new App. It has a Built-in CI/CD and deploy's to Kubernetes with a 'git push' and starts new instances on pull requests.

Thanks for this video! Did you take a look at tanzu application platform for the buy part? It tries to be not too opinionated by allowing to swap pieces. A review of TAP would be really appreciated!

One thing I absolutely LOVE about this video is that it is actually prescriptive. I have watched so many platform engineering videos lately and I haven't seen a single one, except to a small extent Dave Farley's on CD, but other than that they're all idealogical philosophy that emphasize fairly obvious tenets instead of actual implementations.

A terraform module is the exact same thing as a CR and a CRD is exactly the same thing as the golang library backing the terraform module, except unlike the hashicorp public module registry, helm is significantly smaller and less maintained and I have to write the literal CRD code.

Look I agree with what you're saying in principal, but I fail to see how your CRD argument fundamentally differs from taking a bunch of already-written terraform modules from the internet and laying yaml over them? That's just like CRDs + CRs + operators, but all the work is already done. Then you can push the yaml to git and CI converts the yaml to json and the json to hcl and then runs terraform. Now I don't really like this approach, but I'm merely providing a counterpoint here that although the kubernetes api as backend idp idea is more modern, it's not new.

To me what would be new is something that actually exposed really awesome IAM, RBAC and a UI. And k8s api does have great RBAC. But it does nothing with IAM nor a UI. Anyway k8s api is not the only cloud agnostic api.

Your videos are the best devops vids on the net.

Wow!! This is the best. Thank you!

Thank you very much from South Korea. And I have one question. How much of freedom should be given to developers? Should they access to all the tools(eg, Jenkins, Harbor..) behind IDP or they just need access to IDP and SCM?

If I give access to tools under the IDP, their freedom is maximum but it seems to me little hard to manage. What is your suggestion?

Aren't you going to need an IDP to manage and deploy your IDP?

I have no words to describe how I needed this video. IDP is something I want to build but I have no idea, until now, from where I should begin. Thanks for share you knowledge.

Thanks!

Can a beginner be a platform engineer ?

Thank you so much for this video, very easy to follow, straight to the point and helped me accommodate all the concepts correctly in my head.

You just gave me a reason to learn more about CRDs
Your channel has been super helpful 🔥🔥

Brilliant

This is pure gold. Very informative, dense and helpful. I'll have to watch it a couple more times to absorb everything.

Thank you! Awesome video!
Very detailed coverage on process and getting from desired state to actual state. Does it have a part 2 which discuss in details on observing the actual state?

I think you could give more attention to the RBAC part. I totally agree with you that Git is the way to go for business days. But when we think in a crisis scenario, sometimes we get our hands dirty and take action making patches in production.

And add a developer portal on top! With e.g. backstage

Code Rant: Resource Provisioning Complexity Clock

12 o'clock: Clicking buttons in the AWS Management Console
3 o'clock: Calling APIs
6 o'clock: DSL
9 o'clock: GitOps
12 o'clock: Clicking buttons in an internal re-write of the AWS Management Console except now CloudTrail pushes its changes to Git to trigger CI/CD on infrastructure.

Such a great way to look at current landscape of devops. I feel enlightened by this content

My problem is, that everything seems to be about cloud. How do you establish an IDP for on-prem infrastructure like Shares, DNS (Windows Server DNS, BIND), Backup etc.? AFAIK you would have to write your own crossplane providers for each infrastructure piece, or am I missing something?

Thank you, but how crd and kubernetes can do deployments for something else than containers ? i don't figure it out.

Thank you for this videos! We have been building that type of platform internally and didn't have a name for it until now! Amazing content!

Hey Viktor, you've mentioned "numerous studies that show IDPs increase productivity often drastically". Would you happen to have some to share? If so, that would be awesome because it would help to support this idea in future discussions.

DevOps is about FAR MORE than this.
This is only a third of the focus of DevOps.
Coming from the dev side myself I recognise this myopic view of DevOps as dev centric but it is very badly wrong.

Thanks Viktor, a really good video 👍

What you are saying is that every DevOps Engineer should specialize on one stage of what DevOps means and then build teams of specialized people that combine that knowledge and build a strong IDP either for existing projects or a template kind of IDP that could be molded to any sort of project and would make DevOps obsolete. Have I understood anything?

Unfortunate name, IDP. IdP is already taken for "Identity Provider". We definitely need a name for it, though. IDOP (Internal Developer & Operator Platform)? or IDDP (Internal Development and Delivery Platform)?

I like your energy and mindset.
"Define and implement a tool for developers to self-service themselves"
I would love to hear your opinion on what is awaiting us in the next 10 years

Overall, I love most of the ideas in here. However, the disadvantage with this approach is that it discourages new ideas by locking into an approved platform. At least, that's how it's usually done. The IDP here needs to be extensible with very clear information on how to do that and an inclusive ownership team that allows it. Gatekeeping is very easy in this world.

This is incredible, thank you

Thanks!

Is GitLab an example of IDP? From your video, it makes me feel that an IDP is the core platform that control everything the developers and the operation team are using. Am I right? Thanks!

OpenShift is a good example that provides an easy enough to use/understand IDP

I've been developing an IDP for our engineers and I didn't have a word to call it. Having you articulate it in this way is extremely valuable. Thank you!

Hi Viktor!
I'm working on a (golang) cli tools which should be an opensource platform for devops teams to build an IDP on top of it.
Shortly explained, you can define a "repository" which the k8s.yaml will be placed, An "application" which is the logic unit for a service (still thinking about it).
After that you could create any defined resource (pipeline, k8s yamls, etc..) to this defined application in the given repository.
the tool only manage an "environment" repository which then expects ArgoCD/Flux to enforce on the cluster.

Devops teams can define their own resources (templates), and developers can easily use them, all through the cli.

What do you think about the idea ? My intentions is the give a general and easy base for an IDP

A lot on my mind right now.

This so much this. been doing this for years and seem to be the only person in a rather large team doing this. Specifically I wrote an IDP around gitlab-ci to automatically create pipelines :D