The Project Management Institute PMI is the premier standards and professional organization for project management. It works in collaboration with American National Standards Institute (ANSI) and the International Organization for Standardization (ISO) to develop and promote project management standards. Its certifications recognize knowledge and competency and the Project Management Professional (PMP® ) certification is widely considered as the gold standard. PMI has been criticised for being slow to adapt to changes e.g. acknowledging that agile methods were being increasingly adopted in the tech industry as well as these being particularly suited to changing requirements and initiatives with uncertainties e.g. those in startups. However there’s not much generalized awareness in tech about standards beyond those related directly with tech.
Disruption has gone mainstream. Ever since the media picked up on the disruption thread and the technology revolution where startups have disrupted established slow-moving corporations, there has been a flurry of activity at these staid organizations to get onto the innovation bandwagon. Advisory firm Gartner, who gave us “hype cycles” and who thrive on publishing “magic quadrants” has coined the term “bimodal” to sanctify an exploratory, experimental approach to IT. Organizations are rushing to instill “intrapreneurship”, accelerators, corporate venture capital funds, and innovation labs to preempt being disrupted.
With the increasing adoption of agile, there’s been a lot of talk about moving from the project-centric delivery to product-centric delivery model. Initially software/IT borrowed project management practices mainly from manufacturing / construction industries, and such projects generally used sequential, waterfall processes. The technology revolution and the pace of change in IT has made such approaches difficult to sustain and the software industry has been quick to adopt agile methods like scrum which are adapted to product-centric delivery instead of projects.
The last few years has seen a massive change in the data landscape. With the rise of big data, there’s been rapid innovation in the tools, skills and roles working on data systems. Data architectures have evolved beyond monolithic, centralized databases and unwieldy analytic applications to distributed, scalable architectures with simpler collaborative and interactive analytic tools. In this post, I look at the defining features of modern data architectures.
Modern data architectures generally feature the following (though not all of these may be present in the same system):
Over the past few years, Kafka has become the most exciting new addition in the big data distributed architecture. Originally developed at LinkedIn, its founders Jay Kreps, Jun Rao and Neha Narkhede have launched a company Confluent to develop its open-core business model. The software at its core, Apache Kafka reinvents the database log to provide a highly scalable and fault tolerant, high performance distributed system, which serves as the data pipeline backbone for stream data processing.
