Pranitha Nekkanti

Key Features:

• Data Ingestion: Uploaded the YouTube dataset from Kaggle into an S3 bucket with folders for JSON data and region-specific CSV files.
• Data Transformation: Combined data from CSV and JSON files using the common id and category_id columns, and converted them to Parquet format for efficient querying.
• Data Processing with AWS:
  • AWS Lambda: Used Python scripts to convert JSON files to Parquet.
  • AWS Glue: Employed Glue's auto-generated Spark scripts to convert CSV files to Parquet.
  • AWS Athena: Used for SQL querying and joining the cleansed Parquet datasets.
• Automated Workflow: Set up Lambda triggers to automatically execute the ETL pipeline when new data is added to S3.
• Visualization: Visualized the cleansed data using Amazon QuickSight, creating interactive dashboards and visual reports.

Outcomes and Benefits:

• Efficient Data Transformation: Parquet format provided faster queries and reduced storage costs, improving the overall performance of data processing and analysis.
• Automated ETL Pipeline: Lambda triggers automated the processing of new datasets, ensuring real-time availability for querying.
• Improved Data Access: AWS Athena allowed seamless querying of large datasets stored in S3 without needing an extensive ETL infrastructure.
• Actionable Insights: QuickSight dashboards provided real-time insights into YouTube trending data, facilitating better data-driven decision-making.

Technical Details:

• Dataset: YouTube trending dataset from Kaggle, stored as JSON and region-specific CSV files in an S3 bucket.
• Data Transformation:
  • AWS Lambda: Used to convert JSON files to Parquet format.
  • AWS Glue: Employed Glue-generated Python Spark scripts to convert CSV files to Parquet.
• Joining Process: The Parquet outputs from JSON and CSV conversions were joined in AWS Athena using common keys (id, category_id), with type casting to ensure compatibility.
• Visualization: Amazon QuickSight was used for building visual dashboards based on the processed data, enabling users to explore insights interactively.

Conclusion:

This project successfully automated the process of ingesting, transforming, and visualizing YouTube trending data using AWS services. By converting large datasets into Parquet format, it enabled faster querying in Athena and real-time visualization in QuickSight. The combination of AWS Glue, Lambda, and S3 provided a scalable and efficient infrastructure, making it easy to manage and process future datasets. Future enhancements could involve integrating machine learning models to predict trends or further optimizing the data pipeline for more complex analytics.

Key Achievements:

• Designed and implemented a system to monitor vital health parameters including temperature, blood pressure, heart rate, ECG, and body posture.
• Integrated Arduino for sensor interfacing, Raspberry Pi for data processing, and Detectron2 for advanced posture analysis.
• Incorporated ThingSpeak Cloud for real-time data visualization, analysis, and alert generation.
• Developed a user-friendly mobile application for caregivers to access health data and receive alerts.
• Implemented GSM module for secure data transmission, ensuring patient privacy.
• Created a scalable architecture capable of continuous long-term monitoring.

Technologies Used:

Arduino, Raspberry Pi, Detectron2, ThingSpeak Cloud, IoT sensors, GSM module

Impact:

• Awarded best hardware model by the department, recognizing its innovation and practical application.
• Demonstrated potential for significant cost reduction in elderly care by enabling remote monitoring and early intervention.
• Improved quality of life for elderly individuals by allowing them to stay in their homes while receiving continuous health monitoring.

This project showcases my ability to lead a team, integrate complex technologies, and develop practical solutions that address real-world healthcare challenges using IoT and cloud technologies.

Key Features:

• Bulk Data Handling: The system allows for the bulk entry of candidate details, equipped with robust validation mechanisms to ensure data accuracy and integrity.
• Automated Emails: At every stage of the hiring process, automated emails are sent to candidates, keeping them informed and engaged.
• Country-Specific Processes: The hiring process is tailored to meet the specific requirements and regulations of different countries, ensuring compliance and relevance.
• Approval and Verification: The solution includes a structured workflow for approvals and verifications, making sure all necessary checks are in place before proceeding.
• Tracking and Management: The system tracks all stages of the hiring process, providing real-time updates and insights.

Outcomes and Benefits:

• Efficiency and Speed: The fully automated process significantly reduces the turnaround time (TAT) of hiring, allowing for quicker onboarding of candidates.
• Accuracy and Compliance: With automated validations and country-specific processes, the system ensures high accuracy and compliance with local laws and regulations.
• Enhanced Communication: Automated emails at each stage of the process keep candidates informed, improving their experience and engagement.
• Scalability: The solution is designed to handle bulk uploads and manage large volumes of candidate data effortlessly.
• Approval Workflow: Streamlined approval processes and verification steps ensure that all necessary checks are completed systematically.

Technical Details:

• Technology Stack: The project utilizes a robust technology stack, including modern web frameworks, database management systems, and email automation tools.
• Integration: Seamless integration with existing HR systems and databases ensures a smooth transition and adoption.
• Security: Strong data security measures are implemented to protect sensitive candidate information.

Conclusion:

The SF Automation project has successfully transformed the hiring process, making it more efficient, accurate, and user-friendly. By automating various stages and ensuring compliance with country-specific regulations, this solution stands out as a critical tool for modern recruitment.

Key Features:

• Data Migration: Efficient migration of data between cloud and on-premises systems using Azure Data Factory, ensuring minimal disruption and data integrity.
• Pipeline Construction: Building robust pipelines in ADF to automate the data migration process, enabling scheduled and real-time data transfers.
• Data Preprocessing: Cleaning and preprocessing data within Azure Data Factory to ensure data quality. This includes handling missing values and ensuring the data is ready for use.
• Incremental Data Addition: Pipelines in ADF are designed to handle incremental data updates, ensuring that new and modified data is accurately captured and transferred.
• Backend Database: Using SQL Server Management Studio (SSMS) as the backend database to store and manage the migrated data, providing a reliable and scalable storage solution.

Outcomes and Benefits:

• Efficient Data Migration: Streamlined migration process reduces downtime and ensures data integrity during the transfer between cloud and on-premises environments.
• Improved Data Quality: Preprocessing steps like cleaning and adding missing values ensure that the data is accurate, complete, and ready for analysis or operational use.
• Automated Workflows: Automated pipelines reduce manual intervention, minimizing errors and increasing efficiency in data management tasks.
• Scalability: The system is designed to handle large volumes of data and scale with the growing needs of the organization.
• Real-Time Updates: Incremental data addition ensures that the system is always up-to-date with the latest data changes, supporting real-time decision-making.

Technical Details:

• Technology Stack: The project utilizes Azure Data Factory for data migration and pipeline construction, along with SQL Server Management Studio (SSMS) for backend database management.
• Data Processing: Data cleaning, handling missing values, and other preprocessing steps are performed within Azure Data Factory before data storage.
• Integration: Seamless integration with existing data systems and workflows ensures minimal disruption and ease of adoption.
• Security: Robust security measures are implemented to protect data during migration and storage, adhering to best practices and compliance standards.

Conclusion:

The Innovative Aftermarket System project effectively addresses the challenges of data migration between cloud and on-premises environments. By leveraging Azure Data Factory and SSMS, the project ensures efficient, secure, and accurate data transfer and management. The inclusion of data preprocessing steps further enhances the quality and usability of the data.