DeveloperWeek New York 2019

History of Products in the Largest School System in the World We have a lot of legacy applications maintaining data for the past 40-50 years. Over the period we moved a lot of manually processes to automated processes creating more burdens and data volume to our legacy system, keep in mind we have to maintain data for many year at one time. Performance and fast deployment were becoming key factors. Staring from 2012 we took a step back and re-creating our architecture and products. My team supports all Financial and HR transaction in the agency. Modernizing Financial Products In 2012 we started evaluating different techniques and methodologies. As we were Microsoft shop in 2012 we trying to user C#,Asp.net for web end, MSMQ were introduced to stream data between different components. SQL server was our main database. This did not help significantly and the bottle neck remains with data retrieval and data processing. In 2014 - We did another round of POC but this time we used node.js with express and Angular, Redis as enterprise message bus and mongoDB as backend data store. To our surprise the performance were greatly improved. Our financial transaction performance were improved by 300% In 2015-18 - We took the POC and started converting into main stream product contains HR and Financial data for the agency. Maintenance becomes easy as previously we are maintain hundreds of tables in SQL where-as all those were converted into handful of collections. Policy changes comes to us on a rapid pace and it’s easy to add more data in the mongoDB without compromising any data layer. Scalability was awesome; room and experimental freedom was key for many architecture. 2018-19 - We started introducing micro services, with streaming in Kafka and connecting Kafka and mongoDB to bring data from all other systems to mongoDB and make this as single source of truth. Also, we are doing POC with spark and mongoDB so that all analytics and data processing can be done from one platform which we can’t achieve if we stick to our old database. Performance We improved 300% improvement in single transaction. We do millions of transaction per day. Data Growth Not to worry about archiving as the mongoDB is now able to handle huge data load for many years Development Productivity Due to introduction of modern technologies our development productivity increase more than 100%