Nov 02 20:11:56.813122 ip-10-0-140-120 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 20:11:56.813137 ip-10-0-140-120 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 20:11:56.813146 ip-10-0-140-120 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 20:11:56.813570 ip-10-0-140-120 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 20:12:06.710218 ip-10-0-140-120 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 20:12:06.710230 ip-10-0-140-120 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a53b8b626e604487a1c5941309010acc -- Nov 02 20:13:49.461376 ip-10-0-140-120 systemd[1]: Starting Kubernetes Kubelet... Nov 02 20:13:49.865379 ip-10-0-140-120 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 20:13:49.865379 ip-10-0-140-120 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 20:13:49.865379 ip-10-0-140-120 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 20:13:49.865379 ip-10-0-140-120 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 20:13:49.870562 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870489 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:13:49.870562 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870561 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:13:49.870562 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870565 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:13:49.870562 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870568 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870572 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870575 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870577 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870580 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870583 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870585 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870588 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870591 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870594 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870597 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870599 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870601 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870604 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870607 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870609 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870612 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870614 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870617 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870620 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:13:49.870719 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870623 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870626 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870629 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870631 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870637 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870640 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870642 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870645 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870647 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870650 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870653 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870656 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870658 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870661 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870664 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870666 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870669 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870672 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870674 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870677 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:13:49.871295 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870680 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870682 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870685 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870687 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870691 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870693 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870696 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870699 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870702 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870704 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870707 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870710 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870712 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870715 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870717 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870720 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870722 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870725 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870727 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870730 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:13:49.871765 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870732 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870735 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870737 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870740 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870742 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870746 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870749 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870751 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870754 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870756 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870759 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870761 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870764 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870766 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870768 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870771 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870773 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870776 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870778 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870780 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:13:49.872257 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870783 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870786 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870788 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870791 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870794 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870798 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870802 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870805 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870808 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870811 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870813 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870817 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870819 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870822 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870824 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870827 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870829 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870832 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:13:49.872760 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870834 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870838 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870842 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870844 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.870847 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871657 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871664 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871667 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871670 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871672 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871675 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871680 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871684 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871687 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871689 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871693 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871695 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871698 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:13:49.873203 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871701 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871704 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871706 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871709 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871711 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871714 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871716 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871719 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871721 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871724 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871726 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871729 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871731 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871734 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871737 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871739 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871742 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871745 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871747 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871750 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:13:49.873642 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871752 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871755 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871757 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871761 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871765 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871769 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871772 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871775 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871777 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871780 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871783 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871787 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871790 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871792 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871795 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871798 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871800 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871802 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871805 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:13:49.874147 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871807 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871810 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871813 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871815 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871817 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871820 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871823 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871826 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871828 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871831 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871833 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871836 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871839 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871841 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871844 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871846 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871849 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871851 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871854 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871857 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:13:49.874600 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871859 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871861 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871864 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871866 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871870 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871873 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871876 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871878 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871880 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871883 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871885 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871888 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871890 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871892 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871895 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871897 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871900 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871902 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871905 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871908 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:13:49.875060 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871910 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871913 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871915 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871918 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871920 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871923 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871925 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871927 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871930 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871932 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871935 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871937 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871939 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.871942 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872496 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872502 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872510 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872513 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872516 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:13:49.875547 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872519 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872522 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872524 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872527 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872529 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872532 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872534 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872537 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872539 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872542 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872544 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872549 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872553 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872555 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872558 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872560 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872564 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872568 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872571 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:13:49.875994 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872573 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872576 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872578 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872581 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872583 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872586 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872589 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872591 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872594 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872596 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872600 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872603 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872606 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872608 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872611 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872614 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872616 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872619 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872621 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:13:49.876461 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872624 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872626 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872629 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872632 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872634 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872638 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872641 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872643 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872646 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872649 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872651 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872653 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872656 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872659 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872661 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872664 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872666 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872669 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872673 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:13:49.876901 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872676 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872679 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872681 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872684 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872688 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872690 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872693 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872696 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872698 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872701 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872703 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872706 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872708 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872711 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872713 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872715 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872718 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872720 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872724 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:13:49.877374 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872727 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872729 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872731 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872734 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872737 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872739 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872742 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872744 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872747 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872749 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872751 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872754 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872756 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872760 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872763 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872765 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872768 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872771 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872774 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872776 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:13:49.877846 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872779 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872782 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872784 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872787 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.872789 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873734 2576 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873744 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873751 2576 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873755 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873759 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873763 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873767 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873772 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873775 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873778 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873781 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873784 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873787 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873790 2576 flags.go:64] FLAG: --cgroup-root="" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873793 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873796 2576 flags.go:64] FLAG: --client-ca-file="" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873799 2576 flags.go:64] FLAG: --cloud-provider="external" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873801 2576 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873805 2576 flags.go:64] FLAG: --cluster-domain="" Nov 02 20:13:49.878334 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873808 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873819 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873823 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873827 2576 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873831 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873834 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873837 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873840 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873843 2576 flags.go:64] FLAG: --contention-profiling="false" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873846 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873849 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873852 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873855 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873860 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873862 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873865 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873868 2576 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873871 2576 flags.go:64] FLAG: --enable-server="true" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873874 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873879 2576 flags.go:64] FLAG: --event-burst="100" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873882 2576 flags.go:64] FLAG: --event-qps="50" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873885 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873888 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873891 2576 flags.go:64] FLAG: --eviction-hard="" Nov 02 20:13:49.878881 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873895 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873898 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873901 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873904 2576 flags.go:64] FLAG: --eviction-soft="" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873907 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873909 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873912 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873915 2576 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873918 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873921 2576 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873924 2576 flags.go:64] FLAG: --feature-gates="" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873927 2576 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873930 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873934 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873937 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873940 2576 flags.go:64] FLAG: --healthz-port="10248" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873943 2576 flags.go:64] FLAG: --help="false" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873946 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873950 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873953 2576 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873956 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873959 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873962 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 20:13:49.879522 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873965 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873968 2576 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873971 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873974 2576 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873977 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873980 2576 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873982 2576 flags.go:64] FLAG: --kube-reserved="" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873985 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873988 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873991 2576 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873994 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873997 2576 flags.go:64] FLAG: --lock-file="" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.873999 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874002 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874005 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874010 2576 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874013 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874016 2576 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874019 2576 flags.go:64] FLAG: --logging-format="text" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874022 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874025 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874028 2576 flags.go:64] FLAG: --manifest-url="" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874030 2576 flags.go:64] FLAG: --manifest-url-header="" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874035 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874038 2576 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 20:13:49.880062 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874041 2576 flags.go:64] FLAG: --max-pods="110" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874044 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874047 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874050 2576 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874053 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874056 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874059 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874062 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874069 2576 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874072 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874075 2576 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874093 2576 flags.go:64] FLAG: --pod-cidr="" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874097 2576 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874099 2576 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874104 2576 flags.go:64] FLAG: --pods-per-core="0" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874107 2576 flags.go:64] FLAG: --port="10250" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874110 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874113 2576 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0ab8ed626e5ec6847" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874116 2576 flags.go:64] FLAG: --qos-reserved="" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874119 2576 flags.go:64] FLAG: --read-only-port="10255" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874122 2576 flags.go:64] FLAG: --register-node="true" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874125 2576 flags.go:64] FLAG: --register-with-taints="" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874132 2576 flags.go:64] FLAG: --registry-burst="10" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874134 2576 flags.go:64] FLAG: --registry-qps="5" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874137 2576 flags.go:64] FLAG: --reserved-cpus="" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874140 2576 flags.go:64] FLAG: --reserved-memory="" Nov 02 20:13:49.880681 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874143 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874146 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874149 2576 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874152 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874155 2576 flags.go:64] FLAG: --runonce="false" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874158 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874161 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874164 2576 flags.go:64] FLAG: --seccomp-default="false" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874167 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874169 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874173 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874176 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874179 2576 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874182 2576 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874185 2576 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874187 2576 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874192 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874195 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874198 2576 flags.go:64] FLAG: --system-cgroups="" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874201 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874207 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874210 2576 flags.go:64] FLAG: --tls-cert-file="" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874213 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874217 2576 flags.go:64] FLAG: --tls-min-version="" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874219 2576 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 20:13:49.881336 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874222 2576 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874225 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874228 2576 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874231 2576 flags.go:64] FLAG: --v="2" Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874235 2576 flags.go:64] FLAG: --version="false" Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874239 2576 flags.go:64] FLAG: --vmodule="" Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874243 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874246 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874422 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874426 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874429 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874433 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874437 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874441 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874444 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874447 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874450 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874454 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874458 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874461 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874464 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:13:49.881952 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874466 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874469 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874473 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874476 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874478 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874481 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874485 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874487 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874490 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874493 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874495 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874497 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874500 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874502 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874505 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874507 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874510 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874512 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874515 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:13:49.882457 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874517 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874520 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874522 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874525 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874527 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874530 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874532 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874535 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874537 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874540 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874542 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874545 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874548 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874550 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874553 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874557 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874560 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874562 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874565 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874569 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:13:49.882911 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874572 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874574 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874577 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874579 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874582 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874584 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874587 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874589 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874592 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874594 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874597 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874599 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874602 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874605 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874607 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874610 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874612 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874615 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874617 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874619 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:13:49.883469 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874622 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874624 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874626 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874629 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874632 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874634 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874637 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874640 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874643 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874645 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874648 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874651 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874654 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874656 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874659 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874661 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874663 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874666 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874668 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874671 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:13:49.883927 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874673 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:13:49.884407 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874675 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:13:49.884407 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874678 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:13:49.884407 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874680 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:13:49.884407 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874683 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:13:49.884407 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874685 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:13:49.884407 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874687 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:13:49.884407 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874690 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:13:49.884407 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874692 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:13:49.884407 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874694 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:13:49.884407 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874697 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:13:49.884407 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874699 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:13:49.884407 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874701 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:13:49.884407 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874704 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:13:49.884407 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.874729 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 20:13:49.884407 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874764 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:13:49.884407 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874768 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874771 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874774 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874776 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874780 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874782 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874785 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874788 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874791 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874793 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874796 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874798 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874801 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874803 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874806 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874808 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874811 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874814 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874817 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:13:49.884776 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874821 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874824 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874827 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874830 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874832 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874836 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874839 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874842 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874845 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874847 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874849 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874852 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874855 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874857 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874860 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874862 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874865 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874869 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874871 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:13:49.885242 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874874 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874878 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874880 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874883 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874885 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874888 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874890 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874893 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874895 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874898 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874901 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874903 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874905 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874908 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874910 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874913 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874915 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874918 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874920 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:13:49.885695 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874923 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874925 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874928 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874931 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874933 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874936 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874938 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874941 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874943 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874946 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874949 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874952 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874955 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874957 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874961 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874963 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874966 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874969 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874971 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:13:49.886157 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874973 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874976 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874978 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874981 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874983 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874986 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874988 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874991 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874993 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874996 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.874998 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875001 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875003 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875006 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875008 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875010 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875013 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875015 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875018 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875020 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:13:49.886636 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875022 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875025 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875027 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875030 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875033 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875036 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875038 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875042 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.875044 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.881941 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.881956 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.881973 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.881978 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882047 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882053 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882056 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882059 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882062 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882065 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:13:49.887162 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882068 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882070 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882073 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882076 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882095 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882100 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882104 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882108 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882112 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882114 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882117 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882120 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882122 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882125 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882128 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882130 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882133 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882136 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882138 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:13:49.887635 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882141 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882144 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882146 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882149 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882153 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882156 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882158 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882161 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882164 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882166 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882169 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882171 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882173 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882176 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882178 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882181 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882183 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882186 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882188 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882190 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:13:49.888065 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882193 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882195 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882198 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882201 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882203 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882205 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882208 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882210 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882213 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882215 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882218 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882220 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882222 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882225 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882227 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882231 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882234 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882237 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882239 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:13:49.888541 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882242 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882246 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882250 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882252 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882255 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882257 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882260 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882262 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882265 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882267 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882270 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882272 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882276 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882281 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882284 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882286 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882289 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882292 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882294 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:13:49.888972 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882297 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882299 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882302 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882304 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882307 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882309 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882311 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882314 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882316 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882319 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882321 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882324 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882327 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882329 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882332 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882334 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882337 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882339 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882341 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882344 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:13:49.889438 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882346 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882349 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882351 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.882379 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882515 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882520 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882522 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882525 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882527 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882529 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882532 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882535 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882537 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882540 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882542 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882545 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882547 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:13:49.890154 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882550 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882552 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882555 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882558 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882560 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882563 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882566 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882569 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882571 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882574 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882577 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882579 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882582 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882584 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882587 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882589 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882592 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882594 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882597 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:13:49.890711 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882599 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882601 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882604 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882606 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882609 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882611 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882613 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882616 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882618 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882621 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882623 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882625 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882628 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882630 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882633 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882635 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882638 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882640 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882643 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:13:49.891183 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882646 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882648 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882651 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882653 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882656 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882658 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882661 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882663 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882666 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882668 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882670 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882673 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882675 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882677 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882680 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882682 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882685 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882687 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882690 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882692 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:13:49.891626 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882695 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882697 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882699 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882702 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882709 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882711 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882714 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882716 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882719 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882721 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882724 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882727 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882731 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882736 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882740 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882742 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882746 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882748 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882751 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:13:49.892134 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882753 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:13:49.892609 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882756 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:13:49.892609 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882758 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:13:49.892609 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882761 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:13:49.892609 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882763 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:13:49.892609 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882766 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:13:49.892609 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882768 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:13:49.892609 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882771 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:13:49.892609 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882773 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:13:49.892609 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882776 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:13:49.892609 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882778 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:13:49.892609 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882780 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:13:49.892609 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882783 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:13:49.892609 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882785 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:13:49.892609 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882788 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:13:49.892609 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882790 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:13:49.892979 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.882814 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 20:13:49.892979 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882844 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:13:49.892979 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882847 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:13:49.892979 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882849 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:13:49.892979 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882852 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:13:49.892979 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882854 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:13:49.892979 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882857 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:13:49.892979 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882859 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:13:49.892979 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882862 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:13:49.892979 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882864 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:13:49.892979 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882867 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:13:49.892979 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882869 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:13:49.892979 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882871 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:13:49.892979 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882874 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:13:49.892979 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882877 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:13:49.892979 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882880 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882885 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882888 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882891 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882894 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882896 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882898 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882901 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882903 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882906 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882910 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882913 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882916 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882918 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882921 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882924 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882926 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882928 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:13:49.893458 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882931 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882933 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882936 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882938 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882941 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882943 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882945 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882948 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882951 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882953 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882955 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882958 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882960 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882962 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882965 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882968 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882970 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882973 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882975 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882978 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:13:49.894006 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882981 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882983 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882985 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882988 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882992 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882994 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882996 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.882999 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883001 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883004 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883006 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883008 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883011 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883013 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883016 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883018 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883020 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883022 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883025 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:13:49.894509 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883027 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883030 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883032 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883034 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883037 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883039 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883042 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883044 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883047 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883049 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883052 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883054 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883057 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883059 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883061 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883064 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883066 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883070 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883073 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:13:49.894971 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883075 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883091 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883094 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883096 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883099 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883101 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883104 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883106 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883108 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883111 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883113 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883115 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883118 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883120 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883123 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:49.883125 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.883182 2576 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.885391 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.886327 2576 server.go:1014] "Starting client certificate rotation" Nov 02 20:13:49.895691 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.886421 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 20:13:49.896154 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.887193 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 20:13:49.896154 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.888478 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 20:13:49.896154 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.893276 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 20:13:49.905691 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.905674 2576 log.go:25] "Validated CRI v1 runtime API" Nov 02 20:13:49.909339 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.909322 2576 log.go:25] "Validated CRI v1 image API" Nov 02 20:13:49.910262 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.910231 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 20:13:49.915727 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.915708 2576 fs.go:135] Filesystem UUIDs: map[01f4dad2-c4e6-4e5e-aa91-bb591973efe6:/dev/nvme0n1p4 19bdc55b-ed72-4c68-8cc1-fef76173ac9b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 02 20:13:49.915790 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.915727 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 20:13:49.920758 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.920643 2576 manager.go:217] Machine: {Timestamp:2026-11-02 20:13:49.919318936 +0000 UTC m=+0.331454632 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3184079 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29165ebcaa2f561399757821b384f2 SystemUUID:ec29165e-bcaa-2f56-1399-757821b384f2 BootID:a53b8b62-6e60-4487-a1c5-941309010acc Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c1:17:f7:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c1:17:f7:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:fa:0f:c3:bb:78 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 20:13:49.920758 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.920758 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 20:13:49.920857 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.920847 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 20:13:49.923391 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.923371 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 20:13:49.923554 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.923392 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-120.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 20:13:49.923599 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.923557 2576 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 20:13:49.923599 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.923565 2576 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 20:13:49.923652 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.923601 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 20:13:49.924546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.924536 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 20:13:49.925663 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.925653 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 20:13:49.929886 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.929875 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 20:13:49.929990 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.929981 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 20:13:49.930834 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.930824 2576 kubelet.go:493] "Attempting to sync node with API server" Nov 02 20:13:49.930867 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.930837 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 20:13:49.930867 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.930847 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 20:13:49.930867 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.930858 2576 kubelet.go:405] "Adding apiserver pod source" Nov 02 20:13:49.930867 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.930865 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 20:13:49.931162 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.931152 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 20:13:49.933261 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.933238 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 20:13:49.940043 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.940029 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 20:13:49.940922 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.940911 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 20:13:49.940966 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.940941 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 20:13:49.941099 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.941075 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 20:13:49.941131 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.941103 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 20:13:49.941131 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.941110 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 20:13:49.941131 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.941115 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 20:13:49.941131 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.941121 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 20:13:49.941131 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.941127 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 20:13:49.941705 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.941698 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 20:13:49.941736 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.941706 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 20:13:49.941736 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.941714 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 20:13:49.941736 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.941720 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 20:13:49.941736 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.941728 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 20:13:49.941736 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.941737 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 20:13:49.943110 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.943101 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 20:13:49.943110 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.943111 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 20:13:49.946770 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.946751 2576 server.go:1263] "Started kubelet" Nov 02 20:13:49.946897 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.946843 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 20:13:49.946954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.946882 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 20:13:49.946954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.946918 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 20:13:49.950642 ip-10-0-140-120 systemd[1]: Started Kubernetes Kubelet. Nov 02 20:13:49.950878 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.950688 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 20:13:49.953247 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.953232 2576 server.go:317] "Adding debug handlers to kubelet server" Nov 02 20:13:49.958064 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:49.958045 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 20:13:49.960048 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.960032 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 20:13:49.960634 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.960620 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 20:13:49.960722 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:49.960708 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-120.us-west-2.compute.internal\" not found" Nov 02 20:13:49.961389 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.961372 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 20:13:49.961389 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.961389 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 20:13:49.961508 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.961375 2576 factory.go:55] Registering systemd factory Nov 02 20:13:49.961508 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.961405 2576 factory.go:223] Registration of the systemd container factory successfully Nov 02 20:13:49.961508 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.961374 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 20:13:49.961508 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.961492 2576 reconstruct.go:97] "Volume reconstruction finished" Nov 02 20:13:49.961508 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.961497 2576 reconciler.go:29] "Reconciler: start to sync state" Nov 02 20:13:49.961660 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.961589 2576 factory.go:153] Registering CRI-O factory Nov 02 20:13:49.961660 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.961598 2576 factory.go:223] Registration of the crio container factory successfully Nov 02 20:13:49.961723 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.961672 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 20:13:49.961723 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.961687 2576 factory.go:103] Registering Raw factory Nov 02 20:13:49.961723 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.961697 2576 manager.go:1199] Started watching for new ooms in manager Nov 02 20:13:49.962690 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.962676 2576 manager.go:319] Starting recovery of all containers Nov 02 20:13:49.973692 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.973674 2576 manager.go:324] Recovery completed Nov 02 20:13:49.975265 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:49.975185 2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": readdirent /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 02 20:13:49.978152 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.978140 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:13:49.979895 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.979877 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:13:49.979997 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.979912 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:13:49.979997 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.979927 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:13:49.980514 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.980500 2576 cpu_manager.go:226] "Starting" policy="none" Nov 02 20:13:49.980592 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.980515 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 20:13:49.980592 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.980503 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:13:49.980592 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.980529 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 20:13:49.981270 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.981244 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:13:49.981323 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.981275 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:13:49.981323 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.981286 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:13:49.982575 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.982559 2576 policy_none.go:50] "Start" Nov 02 20:13:49.982628 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.982594 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 20:13:49.982628 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.982607 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 20:13:49.985009 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:49.984998 2576 policy_none.go:44] "Start" Nov 02 20:13:50.022539 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.022510 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 20:13:50.048298 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.024888 2576 manager.go:341] "Starting Device Plugin manager" Nov 02 20:13:50.048298 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:50.024920 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 20:13:50.048298 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.024932 2576 server.go:87] "Starting device plugin registration server" Nov 02 20:13:50.048298 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.025177 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 20:13:50.048298 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.025188 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 20:13:50.048298 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.025388 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 20:13:50.048298 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.025453 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 20:13:50.048298 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.025460 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 20:13:50.048298 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:50.026000 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 20:13:50.048298 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:50.026037 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-120.us-west-2.compute.internal\" not found" Nov 02 20:13:50.048298 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.041777 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 20:13:50.048298 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.043033 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 20:13:50.048298 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.043057 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 20:13:50.048298 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.043100 2576 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 20:13:50.048298 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:50.043149 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 20:13:50.086489 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:50.086463 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-120.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 20:13:50.086489 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.086476 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-120.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 20:13:50.125483 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.125425 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:13:50.126218 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.126200 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:13:50.126282 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.126230 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:13:50.126282 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.126240 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:13:50.126282 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.126261 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.132960 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:50.132104 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-120.us-west-2.compute.internal.18e4539018339041 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-120.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-120.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 20:13:49.946728513 +0000 UTC m=+0.358864209,LastTimestamp:2026-11-02 20:13:49.946728513 +0000 UTC m=+0.358864209,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-120.us-west-2.compute.internal,}" Nov 02 20:13:50.143370 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.143345 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal"] Nov 02 20:13:50.143422 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.143405 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:13:50.144154 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.144139 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:13:50.144199 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.144165 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:13:50.144199 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.144176 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:13:50.146223 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.146211 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:13:50.146368 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.146355 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.146402 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.146382 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:13:50.146905 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.146887 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:13:50.146905 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.146902 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:13:50.146996 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.146919 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:13:50.146996 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.146922 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:13:50.146996 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.146935 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:13:50.147098 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.146936 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:13:50.148916 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.148899 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.148993 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.148930 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:13:50.149645 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.149630 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:13:50.149713 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.149652 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:13:50.149713 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.149662 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:13:50.164159 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.164143 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 20:13:50.164239 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.164194 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kclk7" Nov 02 20:13:50.170334 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:50.170317 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-120.us-west-2.compute.internal\" not found" node="ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.172223 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:50.172209 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-120.us-west-2.compute.internal\" not found" node="ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.239571 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.239555 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:13:50.241264 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.241250 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:13:50.262960 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.262939 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b86cdc381e2e2430d60e71836e38622-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"4b86cdc381e2e2430d60e71836e38622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.263037 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.262964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b86cdc381e2e2430d60e71836e38622-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"4b86cdc381e2e2430d60e71836e38622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.263037 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.262986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b918fb29af61c47ab1271b4e381400d1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"b918fb29af61c47ab1271b4e381400d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.334402 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.334382 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:13:50.335950 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.335934 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.336768 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.336753 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:13:50.361547 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.361530 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.363244 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.363224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b86cdc381e2e2430d60e71836e38622-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"4b86cdc381e2e2430d60e71836e38622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.363303 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.363257 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b86cdc381e2e2430d60e71836e38622-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"4b86cdc381e2e2430d60e71836e38622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.363303 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.363274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b918fb29af61c47ab1271b4e381400d1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"b918fb29af61c47ab1271b4e381400d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.363303 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.363233 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b86cdc381e2e2430d60e71836e38622-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"4b86cdc381e2e2430d60e71836e38622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.363413 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.363311 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b918fb29af61c47ab1271b4e381400d1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"b918fb29af61c47ab1271b4e381400d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.363413 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.363341 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b86cdc381e2e2430d60e71836e38622-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"4b86cdc381e2e2430d60e71836e38622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.456933 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.456881 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 20:13:50.457022 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.456986 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.475384 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.475352 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.477544 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.477525 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:13:50.558299 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.558279 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 20:13:50.886355 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.886293 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 20:13:50.886803 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.886445 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 20:13:50.886803 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.886491 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 20:13:50.886803 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:50.886469 2576 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a005d902b89f44e8db632ef0d2b795eb-22154488e9922e7d.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-140-120.us-west-2.compute.internal.18e453901a2da1bd\": read tcp 10.0.140.120:55024->18.190.165.24:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-140-120.us-west-2.compute.internal.18e453901a2da1bd default 4813 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-120.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-120.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-120.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 20:13:49 +0000 UTC,LastTimestamp:2026-11-02 20:13:50.144153442 +0000 UTC m=+0.556289137,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-120.us-west-2.compute.internal,}" Nov 02 20:13:50.932023 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.931993 2576 apiserver.go:52] "Watching apiserver" Nov 02 20:13:50.960489 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:50.960466 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 20:13:51.170414 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.170345 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 20:08:50 +0000 UTC" deadline="2028-06-07 15:49:06.044432562 +0000 UTC" Nov 02 20:13:51.170414 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.170377 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13987h35m14.874058027s" Nov 02 20:13:51.180938 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.180921 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 20:13:51.181234 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.181218 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-jjtbq","kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal","openshift-image-registry/node-ca-7pl28","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-rphxh","kube-system/konnectivity-agent-jbk5q","openshift-dns/node-resolver-26nmv","openshift-multus/multus-additional-cni-plugins-bvfmm","openshift-multus/multus-gxrvt","openshift-network-operator/iptables-alerter-l54nk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm","openshift-cluster-node-tuning-operator/tuned-fv8rs","openshift-multus/network-metrics-daemon-5kzc7","openshift-ovn-kubernetes/ovnkube-node-9ngnp"] Nov 02 20:13:51.183646 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.183632 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-26nmv" Nov 02 20:13:51.185597 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.185575 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:13:51.185686 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.185641 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kzc7" podUID="30d6b618-cc4e-48b8-a7b4-9649a4242aad" Nov 02 20:13:51.187652 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.187635 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:13:51.189688 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.189671 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:13:51.189776 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.189716 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rphxh" podUID="f8dd1189-9c04-4ba1-a349-04e7dabcab52" Nov 02 20:13:51.191754 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.191737 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:13:51.193684 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.193669 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.195675 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.195656 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:51.195764 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.195731 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjtbq" podUID="afe07c4f-7282-4239-9761-c61ef4153cd9" Nov 02 20:13:51.197828 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.197804 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:13:51.200134 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.200119 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.202352 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.202334 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.204674 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.204661 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.206883 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.206868 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.233118 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.233101 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 20:13:51.233118 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.233104 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 20:13:51.233251 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.233179 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kz985\"" Nov 02 20:13:51.236691 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.236670 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 20:13:51.236982 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.236965 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vjrtf\"" Nov 02 20:13:51.236982 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.236974 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 20:13:51.237095 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.236990 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 20:13:51.240977 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.240959 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 20:13:51.241114 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.240996 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 20:13:51.241114 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.241074 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ldshw\"" Nov 02 20:13:51.242856 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.242841 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 20:13:51.243074 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.243061 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 20:13:51.243145 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.243131 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 20:13:51.243199 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.243173 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-t6dvk\"" Nov 02 20:13:51.243274 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.243259 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 20:13:51.247709 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.247685 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 20:13:51.247816 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.247770 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 20:13:51.247877 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.247832 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qfmz2\"" Nov 02 20:13:51.248242 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.248224 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:13:51.250697 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.249396 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 20:13:51.250697 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.249518 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7m5tr\"" Nov 02 20:13:51.262255 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.262240 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 20:13:51.267612 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.267592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-registration-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.267709 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.267616 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e68d28-771c-48e8-ba38-f8aad80a6bf9-tmp-dir\") pod \"node-resolver-26nmv\" (UID: \"39e68d28-771c-48e8-ba38-f8aad80a6bf9\") " pod="openshift-dns/node-resolver-26nmv" Nov 02 20:13:51.267709 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.267632 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcq6w\" (UniqueName: \"kubernetes.io/projected/9fec747e-0efa-40f5-afaf-d45d820700b1-kube-api-access-wcq6w\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.267709 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.267684 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-sys\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.267862 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.267726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf87f\" (UniqueName: \"kubernetes.io/projected/39e68d28-771c-48e8-ba38-f8aad80a6bf9-kube-api-access-jf87f\") pod \"node-resolver-26nmv\" (UID: \"39e68d28-771c-48e8-ba38-f8aad80a6bf9\") " pod="openshift-dns/node-resolver-26nmv" Nov 02 20:13:51.267862 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.267779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-socket-dir-parent\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.267862 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.267836 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-multus-certs\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.267978 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.267894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7nbm\" (UniqueName: \"kubernetes.io/projected/68a10036-5c78-4b01-b06d-075a727a5d0b-kube-api-access-s7nbm\") pod \"iptables-alerter-l54nk\" (UID: \"68a10036-5c78-4b01-b06d-075a727a5d0b\") " pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:13:51.267978 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.267930 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-etc-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.267978 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.267959 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-ovn\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.268070 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268011 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.268070 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-host\") pod \"node-ca-7pl28\" (UID: \"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd\") " pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:13:51.268178 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:51.268211 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-node-log\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.268211 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.268285 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268226 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.268285 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/afe07c4f-7282-4239-9761-c61ef4153cd9-dbus\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:51.268285 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-log-socket\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.268385 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268309 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64f527e7-dcc8-49be-8c15-a078a7aed186-ovn-node-metrics-cert\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.268385 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-run\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.268385 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-cnibin\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.268385 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-etc-selinux\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.268385 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268380 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/afe07c4f-7282-4239-9761-c61ef4153cd9-kubelet-config\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:51.268599 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-kubelet\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.268599 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.268599 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-tuned\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.268599 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvmzm\" (UniqueName: \"kubernetes.io/projected/1e96f673-1cf6-48f1-a55d-347662c75b16-kube-api-access-qvmzm\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.268599 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268526 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-kubelet\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.268599 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268547 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysctl-d\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.268599 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-sys-fs\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.268909 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268614 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-conf-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.268909 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268652 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-modprobe-d\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.268909 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t22cd\" (UniqueName: \"kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd\") pod \"network-check-target-rphxh\" (UID: \"f8dd1189-9c04-4ba1-a349-04e7dabcab52\") " pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:13:51.268909 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-cni-bin\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.268909 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268755 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-run-ovn-kubernetes\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.268909 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-env-overrides\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.268909 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-host\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.268909 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-cni-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.268909 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268847 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-run-netns\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.268909 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268877 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-systemd\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.268909 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b51723af-6a33-4535-91c0-905ce2a51b24-konnectivity-ca\") pod \"konnectivity-agent-jbk5q\" (UID: \"b51723af-6a33-4535-91c0-905ce2a51b24\") " pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:13:51.268909 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268907 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-lib-modules\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.269264 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268954 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snn5k\" (UniqueName: \"kubernetes.io/projected/bcc543bf-8ffd-488d-999a-2b8da64e9c31-kube-api-access-snn5k\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.269264 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-device-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.269264 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.268991 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-cni-netd\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.269264 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269015 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-systemd\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.269264 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269048 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-netns\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.269264 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-etc-kubernetes\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.269264 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269109 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcc543bf-8ffd-488d-999a-2b8da64e9c31-tmp\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.269264 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgpnn\" (UniqueName: \"kubernetes.io/projected/30d6b618-cc4e-48b8-a7b4-9649a4242aad-kube-api-access-bgpnn\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:13:51.269264 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269179 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-serviceca\") pod \"node-ca-7pl28\" (UID: \"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd\") " pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:13:51.269264 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-system-cni-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.269264 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-cnibin\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.269264 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fec747e-0efa-40f5-afaf-d45d820700b1-cni-binary-copy\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.269264 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-hostroot\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.269635 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-daemon-config\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.269635 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-systemd-units\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.269635 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68a10036-5c78-4b01-b06d-075a727a5d0b-iptables-alerter-script\") pod \"iptables-alerter-l54nk\" (UID: \"68a10036-5c78-4b01-b06d-075a727a5d0b\") " pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:13:51.269635 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68a10036-5c78-4b01-b06d-075a727a5d0b-host-slash\") pod \"iptables-alerter-l54nk\" (UID: \"68a10036-5c78-4b01-b06d-075a727a5d0b\") " pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:13:51.269635 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw5x4\" (UniqueName: \"kubernetes.io/projected/64f527e7-dcc8-49be-8c15-a078a7aed186-kube-api-access-gw5x4\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.269635 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:13:51.269635 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269430 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-slash\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.269635 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269465 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-cni-binary-copy\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.269635 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-socket-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.269635 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/39e68d28-771c-48e8-ba38-f8aad80a6bf9-hosts-file\") pod \"node-resolver-26nmv\" (UID: \"39e68d28-771c-48e8-ba38-f8aad80a6bf9\") " pod="openshift-dns/node-resolver-26nmv" Nov 02 20:13:51.269635 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269532 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd99v\" (UniqueName: \"kubernetes.io/projected/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-kube-api-access-hd99v\") pod \"node-ca-7pl28\" (UID: \"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd\") " pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:13:51.269635 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.269635 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269616 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-ovnkube-script-lib\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.270106 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-system-cni-dir\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.270106 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269660 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-k8s-cni-cncf-io\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.270106 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-var-lib-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.270106 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-cni-bin\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.270106 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-ovnkube-config\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.270106 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysconfig\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.270106 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-var-lib-kubelet\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.270106 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.270106 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269776 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-cni-multus\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.270106 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269802 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b51723af-6a33-4535-91c0-905ce2a51b24-agent-certs\") pod \"konnectivity-agent-jbk5q\" (UID: \"b51723af-6a33-4535-91c0-905ce2a51b24\") " pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:13:51.270106 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269831 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-kubernetes\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.270106 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269847 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysctl-conf\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.270106 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.269864 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5mrg\" (UniqueName: \"kubernetes.io/projected/a138b9a8-1f14-42a3-97f6-9e2612f83470-kube-api-access-x5mrg\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.277591 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.277566 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 20:13:51.277701 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.277627 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 20:13:51.277701 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.277657 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 20:13:51.277701 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.277657 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 20:13:51.280648 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.280398 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 20:13:51.280648 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.280432 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 20:13:51.280648 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.280461 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 20:13:51.280648 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.280490 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 20:13:51.280648 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.280580 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 20:13:51.280648 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.280612 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xxqx9\"" Nov 02 20:13:51.280648 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.280642 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 20:13:51.280989 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.280673 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 20:13:51.280989 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.280706 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5vs7j\"" Nov 02 20:13:51.280989 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.280722 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 20:13:51.280989 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.280791 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:13:51.280989 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.280802 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hxslf\"" Nov 02 20:13:51.280989 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.280846 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v2wvh" Nov 02 20:13:51.371012 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.370988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-cni-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.371012 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-run-netns\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.371173 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-systemd\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.371173 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b51723af-6a33-4535-91c0-905ce2a51b24-konnectivity-ca\") pod \"konnectivity-agent-jbk5q\" (UID: \"b51723af-6a33-4535-91c0-905ce2a51b24\") " pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:13:51.371173 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371058 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-lib-modules\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.371173 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371073 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snn5k\" (UniqueName: \"kubernetes.io/projected/bcc543bf-8ffd-488d-999a-2b8da64e9c31-kube-api-access-snn5k\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.371173 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-device-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.371173 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371130 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-run-netns\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.371173 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371141 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-cni-netd\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.371173 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371130 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-cni-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-systemd\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-netns\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371227 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-etc-kubernetes\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371237 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-cni-netd\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcc543bf-8ffd-488d-999a-2b8da64e9c31-tmp\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371263 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-netns\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371240 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-systemd\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgpnn\" (UniqueName: \"kubernetes.io/projected/30d6b618-cc4e-48b8-a7b4-9649a4242aad-kube-api-access-bgpnn\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-serviceca\") pod \"node-ca-7pl28\" (UID: \"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd\") " pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371333 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-device-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371338 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-system-cni-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371345 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-systemd\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371357 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-lib-modules\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-cnibin\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371281 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-etc-kubernetes\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fec747e-0efa-40f5-afaf-d45d820700b1-cni-binary-copy\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-hostroot\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.371546 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371482 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-system-cni-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-daemon-config\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371423 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-cnibin\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-systemd-units\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371539 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68a10036-5c78-4b01-b06d-075a727a5d0b-iptables-alerter-script\") pod \"iptables-alerter-l54nk\" (UID: \"68a10036-5c78-4b01-b06d-075a727a5d0b\") " pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371552 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-hostroot\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68a10036-5c78-4b01-b06d-075a727a5d0b-host-slash\") pod \"iptables-alerter-l54nk\" (UID: \"68a10036-5c78-4b01-b06d-075a727a5d0b\") " pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw5x4\" (UniqueName: \"kubernetes.io/projected/64f527e7-dcc8-49be-8c15-a078a7aed186-kube-api-access-gw5x4\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371614 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.371705 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371744 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b51723af-6a33-4535-91c0-905ce2a51b24-konnectivity-ca\") pod \"konnectivity-agent-jbk5q\" (UID: \"b51723af-6a33-4535-91c0-905ce2a51b24\") " pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.371770 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs podName:30d6b618-cc4e-48b8-a7b4-9649a4242aad nodeName:}" failed. No retries permitted until 2026-11-02 20:13:51.871741864 +0000 UTC m=+2.283877547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs") pod "network-metrics-daemon-5kzc7" (UID: "30d6b618-cc4e-48b8-a7b4-9649a4242aad") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371594 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-systemd-units\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371799 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-serviceca\") pod \"node-ca-7pl28\" (UID: \"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd\") " pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371816 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68a10036-5c78-4b01-b06d-075a727a5d0b-host-slash\") pod \"iptables-alerter-l54nk\" (UID: \"68a10036-5c78-4b01-b06d-075a727a5d0b\") " pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-slash\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-cni-binary-copy\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.372360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371984 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fec747e-0efa-40f5-afaf-d45d820700b1-cni-binary-copy\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.371990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-socket-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372006 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-slash\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/39e68d28-771c-48e8-ba38-f8aad80a6bf9-hosts-file\") pod \"node-resolver-26nmv\" (UID: \"39e68d28-771c-48e8-ba38-f8aad80a6bf9\") " pod="openshift-dns/node-resolver-26nmv" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd99v\" (UniqueName: \"kubernetes.io/projected/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-kube-api-access-hd99v\") pod \"node-ca-7pl28\" (UID: \"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd\") " pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372109 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-socket-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372120 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-ovnkube-script-lib\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-system-cni-dir\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372146 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/39e68d28-771c-48e8-ba38-f8aad80a6bf9-hosts-file\") pod \"node-resolver-26nmv\" (UID: \"39e68d28-771c-48e8-ba38-f8aad80a6bf9\") " pod="openshift-dns/node-resolver-26nmv" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-k8s-cni-cncf-io\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-var-lib-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372197 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68a10036-5c78-4b01-b06d-075a727a5d0b-iptables-alerter-script\") pod \"iptables-alerter-l54nk\" (UID: \"68a10036-5c78-4b01-b06d-075a727a5d0b\") " pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372204 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-cni-bin\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-ovnkube-config\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372237 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-cni-bin\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372257 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysconfig\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.373190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372266 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-system-cni-dir\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372298 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-k8s-cni-cncf-io\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372306 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysconfig\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372331 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-var-lib-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-var-lib-kubelet\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372355 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-daemon-config\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372390 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372417 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-cni-multus\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b51723af-6a33-4535-91c0-905ce2a51b24-agent-certs\") pod \"konnectivity-agent-jbk5q\" (UID: \"b51723af-6a33-4535-91c0-905ce2a51b24\") " pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-kubernetes\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysctl-conf\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372528 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5mrg\" (UniqueName: \"kubernetes.io/projected/a138b9a8-1f14-42a3-97f6-9e2612f83470-kube-api-access-x5mrg\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372541 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372584 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-kubernetes\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372575 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-cni-multus\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372603 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-cni-binary-copy\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372648 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-ovnkube-script-lib\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.373827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-registration-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372728 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-registration-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e68d28-771c-48e8-ba38-f8aad80a6bf9-tmp-dir\") pod \"node-resolver-26nmv\" (UID: \"39e68d28-771c-48e8-ba38-f8aad80a6bf9\") " pod="openshift-dns/node-resolver-26nmv" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372783 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysctl-conf\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcq6w\" (UniqueName: \"kubernetes.io/projected/9fec747e-0efa-40f5-afaf-d45d820700b1-kube-api-access-wcq6w\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372850 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-var-lib-kubelet\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-sys\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372875 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-ovnkube-config\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372896 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-sys\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf87f\" (UniqueName: \"kubernetes.io/projected/39e68d28-771c-48e8-ba38-f8aad80a6bf9-kube-api-access-jf87f\") pod \"node-resolver-26nmv\" (UID: \"39e68d28-771c-48e8-ba38-f8aad80a6bf9\") " pod="openshift-dns/node-resolver-26nmv" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372953 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-socket-dir-parent\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.372988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-multus-certs\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373010 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7nbm\" (UniqueName: \"kubernetes.io/projected/68a10036-5c78-4b01-b06d-075a727a5d0b-kube-api-access-s7nbm\") pod \"iptables-alerter-l54nk\" (UID: \"68a10036-5c78-4b01-b06d-075a727a5d0b\") " pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373031 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e68d28-771c-48e8-ba38-f8aad80a6bf9-tmp-dir\") pod \"node-resolver-26nmv\" (UID: \"39e68d28-771c-48e8-ba38-f8aad80a6bf9\") " pod="openshift-dns/node-resolver-26nmv" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373098 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-multus-certs\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-etc-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373353 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-socket-dir-parent\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.374696 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373377 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-etc-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-ovn\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-host\") pod \"node-ca-7pl28\" (UID: \"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd\") " pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-node-log\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373522 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-ovn\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.373856 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/afe07c4f-7282-4239-9761-c61ef4153cd9-dbus\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.373916 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret podName:afe07c4f-7282-4239-9761-c61ef4153cd9 nodeName:}" failed. No retries permitted until 2026-11-02 20:13:51.873896721 +0000 UTC m=+2.286032418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret") pod "global-pull-secret-syncer-jjtbq" (UID: "afe07c4f-7282-4239-9761-c61ef4153cd9") : object "kube-system"/"original-pull-secret" not registered Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373929 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-log-socket\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373916 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-host\") pod \"node-ca-7pl28\" (UID: \"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd\") " pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373959 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-node-log\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373964 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/afe07c4f-7282-4239-9761-c61ef4153cd9-dbus\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:51.375424 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.373964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64f527e7-dcc8-49be-8c15-a078a7aed186-ovn-node-metrics-cert\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374002 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-log-socket\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-run\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374072 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-run\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-cnibin\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374130 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-cnibin\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-etc-selinux\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/afe07c4f-7282-4239-9761-c61ef4153cd9-kubelet-config\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374201 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374209 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-kubelet\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374226 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-etc-selinux\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374270 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/afe07c4f-7282-4239-9761-c61ef4153cd9-kubelet-config\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374281 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-kubelet\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374305 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-tuned\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374304 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmzm\" (UniqueName: \"kubernetes.io/projected/1e96f673-1cf6-48f1-a55d-347662c75b16-kube-api-access-qvmzm\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.375954 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-kubelet\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysctl-d\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-sys-fs\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374526 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-conf-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374614 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-modprobe-d\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374622 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysctl-d\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374640 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t22cd\" (UniqueName: \"kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd\") pod \"network-check-target-rphxh\" (UID: \"f8dd1189-9c04-4ba1-a349-04e7dabcab52\") " pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374645 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-sys-fs\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374690 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-kubelet\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374773 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-cni-bin\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374778 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-conf-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-run-ovn-kubernetes\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374809 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-modprobe-d\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374831 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374861 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-run-ovn-kubernetes\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374859 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-cni-bin\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-env-overrides\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.376580 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.374957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-host\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.377063 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.375147 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-host\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.377063 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.376112 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcc543bf-8ffd-488d-999a-2b8da64e9c31-tmp\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.377063 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.376232 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-env-overrides\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.377063 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.376391 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b51723af-6a33-4535-91c0-905ce2a51b24-agent-certs\") pod \"konnectivity-agent-jbk5q\" (UID: \"b51723af-6a33-4535-91c0-905ce2a51b24\") " pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:13:51.377063 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.376577 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-tuned\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.377063 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.376792 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64f527e7-dcc8-49be-8c15-a078a7aed186-ovn-node-metrics-cert\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.430875 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.430157 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 20:13:51.430875 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.430196 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 20:13:51.430875 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.430209 2576 projected.go:196] Error preparing data for projected volume kube-api-access-t22cd for pod openshift-network-diagnostics/network-check-target-rphxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:13:51.430875 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.430350 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd podName:f8dd1189-9c04-4ba1-a349-04e7dabcab52 nodeName:}" failed. No retries permitted until 2026-11-02 20:13:51.930325148 +0000 UTC m=+2.342460846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t22cd" (UniqueName: "kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd") pod "network-check-target-rphxh" (UID: "f8dd1189-9c04-4ba1-a349-04e7dabcab52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:13:51.430875 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.430766 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw5x4\" (UniqueName: \"kubernetes.io/projected/64f527e7-dcc8-49be-8c15-a078a7aed186-kube-api-access-gw5x4\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.430875 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.430789 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5mrg\" (UniqueName: \"kubernetes.io/projected/a138b9a8-1f14-42a3-97f6-9e2612f83470-kube-api-access-x5mrg\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.431314 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.431215 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcq6w\" (UniqueName: \"kubernetes.io/projected/9fec747e-0efa-40f5-afaf-d45d820700b1-kube-api-access-wcq6w\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.431533 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.431511 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7nbm\" (UniqueName: \"kubernetes.io/projected/68a10036-5c78-4b01-b06d-075a727a5d0b-kube-api-access-s7nbm\") pod \"iptables-alerter-l54nk\" (UID: \"68a10036-5c78-4b01-b06d-075a727a5d0b\") " pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:13:51.432387 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.432366 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snn5k\" (UniqueName: \"kubernetes.io/projected/bcc543bf-8ffd-488d-999a-2b8da64e9c31-kube-api-access-snn5k\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.432387 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.432383 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf87f\" (UniqueName: \"kubernetes.io/projected/39e68d28-771c-48e8-ba38-f8aad80a6bf9-kube-api-access-jf87f\") pod \"node-resolver-26nmv\" (UID: \"39e68d28-771c-48e8-ba38-f8aad80a6bf9\") " pod="openshift-dns/node-resolver-26nmv" Nov 02 20:13:51.432540 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.432467 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd99v\" (UniqueName: \"kubernetes.io/projected/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-kube-api-access-hd99v\") pod \"node-ca-7pl28\" (UID: \"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd\") " pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:13:51.432540 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.432482 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgpnn\" (UniqueName: \"kubernetes.io/projected/30d6b618-cc4e-48b8-a7b4-9649a4242aad-kube-api-access-bgpnn\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:13:51.433499 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.433474 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmzm\" (UniqueName: \"kubernetes.io/projected/1e96f673-1cf6-48f1-a55d-347662c75b16-kube-api-access-qvmzm\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.512736 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.512706 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-26nmv" Nov 02 20:13:51.530433 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.530411 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:13:51.549930 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.549908 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:13:51.568669 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.568653 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gxrvt" Nov 02 20:13:51.570790 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.570774 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:13:51.579676 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:51.579645 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b86cdc381e2e2430d60e71836e38622.slice/crio-690a5c20ac6d5a4dc1bf8a9ccf0448d9b7d97f3c5d154b65e2b20a6534b579e6 WatchSource:0}: Error finding container 690a5c20ac6d5a4dc1bf8a9ccf0448d9b7d97f3c5d154b65e2b20a6534b579e6: Status 404 returned error can't find the container with id 690a5c20ac6d5a4dc1bf8a9ccf0448d9b7d97f3c5d154b65e2b20a6534b579e6 Nov 02 20:13:51.580609 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:51.580590 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51723af_6a33_4535_91c0_905ce2a51b24.slice/crio-cbc0f8137d19d2acb1fda8e0914da1d28884aabc4d694a37da80cd622155e926 WatchSource:0}: Error finding container cbc0f8137d19d2acb1fda8e0914da1d28884aabc4d694a37da80cd622155e926: Status 404 returned error can't find the container with id cbc0f8137d19d2acb1fda8e0914da1d28884aabc4d694a37da80cd622155e926 Nov 02 20:13:51.581368 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:51.581350 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb918fb29af61c47ab1271b4e381400d1.slice/crio-8ea136b3030a2d7db3d157c0d72d2877085de68f36077b113c39ef0a8d689e27 WatchSource:0}: Error finding container 8ea136b3030a2d7db3d157c0d72d2877085de68f36077b113c39ef0a8d689e27: Status 404 returned error can't find the container with id 8ea136b3030a2d7db3d157c0d72d2877085de68f36077b113c39ef0a8d689e27 Nov 02 20:13:51.582842 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:51.582825 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fec747e_0efa_40f5_afaf_d45d820700b1.slice/crio-7bf224f0387a6c1763335ce89ce4f008842b70867aa913d224958400e5e831fe WatchSource:0}: Error finding container 7bf224f0387a6c1763335ce89ce4f008842b70867aa913d224958400e5e831fe: Status 404 returned error can't find the container with id 7bf224f0387a6c1763335ce89ce4f008842b70867aa913d224958400e5e831fe Nov 02 20:13:51.587306 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.587282 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:13:51.599579 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.599562 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:13:51.608252 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.608233 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:13:51.608471 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:51.608451 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f527e7_dcc8_49be_8c15_a078a7aed186.slice/crio-6c7aeeaa257f53ea9aa8d328684a6c694a927c24739b9dbd19481ec1d8ec2d30 WatchSource:0}: Error finding container 6c7aeeaa257f53ea9aa8d328684a6c694a927c24739b9dbd19481ec1d8ec2d30: Status 404 returned error can't find the container with id 6c7aeeaa257f53ea9aa8d328684a6c694a927c24739b9dbd19481ec1d8ec2d30 Nov 02 20:13:51.615931 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:51.615911 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc543bf_8ffd_488d_999a_2b8da64e9c31.slice/crio-230579ac09e4d4ca344ecf1e87cc153807abd903e9b5ca7827237b8dc13e4dcf WatchSource:0}: Error finding container 230579ac09e4d4ca344ecf1e87cc153807abd903e9b5ca7827237b8dc13e4dcf: Status 404 returned error can't find the container with id 230579ac09e4d4ca344ecf1e87cc153807abd903e9b5ca7827237b8dc13e4dcf Nov 02 20:13:51.642501 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.642479 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:13:51.644601 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.644585 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:13:51.652152 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:51.652134 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e96f673_1cf6_48f1_a55d_347662c75b16.slice/crio-07ab293df918258cd3b264ce3c4eedc9958d2dae6b334a34d3a6e19c5c41af80 WatchSource:0}: Error finding container 07ab293df918258cd3b264ce3c4eedc9958d2dae6b334a34d3a6e19c5c41af80: Status 404 returned error can't find the container with id 07ab293df918258cd3b264ce3c4eedc9958d2dae6b334a34d3a6e19c5c41af80 Nov 02 20:13:51.655758 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:51.655725 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda138b9a8_1f14_42a3_97f6_9e2612f83470.slice/crio-612faca9a4be33f531f8cf9d8762516888219f40744f562cdb9035696360431c WatchSource:0}: Error finding container 612faca9a4be33f531f8cf9d8762516888219f40744f562cdb9035696360431c: Status 404 returned error can't find the container with id 612faca9a4be33f531f8cf9d8762516888219f40744f562cdb9035696360431c Nov 02 20:13:51.877785 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.877752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:51.877897 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.877809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:13:51.877897 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.877870 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 20:13:51.877897 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.877889 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:13:51.878083 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.877936 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs podName:30d6b618-cc4e-48b8-a7b4-9649a4242aad nodeName:}" failed. No retries permitted until 2026-11-02 20:13:52.877922472 +0000 UTC m=+3.290058157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs") pod "network-metrics-daemon-5kzc7" (UID: "30d6b618-cc4e-48b8-a7b4-9649a4242aad") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:13:51.878083 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.877950 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret podName:afe07c4f-7282-4239-9761-c61ef4153cd9 nodeName:}" failed. No retries permitted until 2026-11-02 20:13:52.877944184 +0000 UTC m=+3.290079869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret") pod "global-pull-secret-syncer-jjtbq" (UID: "afe07c4f-7282-4239-9761-c61ef4153cd9") : object "kube-system"/"original-pull-secret" not registered Nov 02 20:13:51.978695 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:51.978635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t22cd\" (UniqueName: \"kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd\") pod \"network-check-target-rphxh\" (UID: \"f8dd1189-9c04-4ba1-a349-04e7dabcab52\") " pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:13:51.979196 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.978730 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 20:13:51.979196 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.978745 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 20:13:51.979196 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.978753 2576 projected.go:196] Error preparing data for projected volume kube-api-access-t22cd for pod openshift-network-diagnostics/network-check-target-rphxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:13:51.979196 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:51.978802 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd podName:f8dd1189-9c04-4ba1-a349-04e7dabcab52 nodeName:}" failed. No retries permitted until 2026-11-02 20:13:52.978790031 +0000 UTC m=+3.390925716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t22cd" (UniqueName: "kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd") pod "network-check-target-rphxh" (UID: "f8dd1189-9c04-4ba1-a349-04e7dabcab52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:13:52.047468 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:52.047424 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvfmm" event={"ID":"a138b9a8-1f14-42a3-97f6-9e2612f83470","Type":"ContainerStarted","Data":"612faca9a4be33f531f8cf9d8762516888219f40744f562cdb9035696360431c"} Nov 02 20:13:52.048637 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:52.048616 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" event={"ID":"bcc543bf-8ffd-488d-999a-2b8da64e9c31","Type":"ContainerStarted","Data":"230579ac09e4d4ca344ecf1e87cc153807abd903e9b5ca7827237b8dc13e4dcf"} Nov 02 20:13:52.049877 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:52.049846 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" event={"ID":"64f527e7-dcc8-49be-8c15-a078a7aed186","Type":"ContainerStarted","Data":"6c7aeeaa257f53ea9aa8d328684a6c694a927c24739b9dbd19481ec1d8ec2d30"} Nov 02 20:13:52.050994 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:52.050970 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l54nk" event={"ID":"68a10036-5c78-4b01-b06d-075a727a5d0b","Type":"ContainerStarted","Data":"15606ef9514cdd2b5c88060ac28788605d80266d1a0ec4a2b215b17077900d8d"} Nov 02 20:13:52.052168 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:52.052150 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jbk5q" event={"ID":"b51723af-6a33-4535-91c0-905ce2a51b24","Type":"ContainerStarted","Data":"cbc0f8137d19d2acb1fda8e0914da1d28884aabc4d694a37da80cd622155e926"} Nov 02 20:13:52.053282 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:52.053264 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" event={"ID":"b918fb29af61c47ab1271b4e381400d1","Type":"ContainerStarted","Data":"8ea136b3030a2d7db3d157c0d72d2877085de68f36077b113c39ef0a8d689e27"} Nov 02 20:13:52.054352 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:52.054320 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" event={"ID":"4b86cdc381e2e2430d60e71836e38622","Type":"ContainerStarted","Data":"690a5c20ac6d5a4dc1bf8a9ccf0448d9b7d97f3c5d154b65e2b20a6534b579e6"} Nov 02 20:13:52.055328 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:52.055311 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" event={"ID":"1e96f673-1cf6-48f1-a55d-347662c75b16","Type":"ContainerStarted","Data":"07ab293df918258cd3b264ce3c4eedc9958d2dae6b334a34d3a6e19c5c41af80"} Nov 02 20:13:52.056380 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:52.056359 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gxrvt" event={"ID":"9fec747e-0efa-40f5-afaf-d45d820700b1","Type":"ContainerStarted","Data":"7bf224f0387a6c1763335ce89ce4f008842b70867aa913d224958400e5e831fe"} Nov 02 20:13:52.159809 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:52.159787 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:13:52.241011 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:52.240965 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:13:52.284217 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:52.284175 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 20:08:51 +0000 UTC" deadline="2028-04-03 12:40:36.185718239 +0000 UTC" Nov 02 20:13:52.284217 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:52.284210 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12424h26m43.901510207s" Nov 02 20:13:52.400055 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:52.400031 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e68d28_771c_48e8_ba38_f8aad80a6bf9.slice/crio-14e005f90d52a2dc6b1bcb210c65223609ecf4699f6910ad0cf71b52d3e38265 WatchSource:0}: Error finding container 14e005f90d52a2dc6b1bcb210c65223609ecf4699f6910ad0cf71b52d3e38265: Status 404 returned error can't find the container with id 14e005f90d52a2dc6b1bcb210c65223609ecf4699f6910ad0cf71b52d3e38265 Nov 02 20:13:52.460880 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:13:52.460858 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a9f726_c51f_4668_a4b3_5eb8ca1df1cd.slice/crio-bc66d97e073c8610d57e2e9b89438cf1dd411d9b8cfa417c158b770ee2cdf1cf WatchSource:0}: Error finding container bc66d97e073c8610d57e2e9b89438cf1dd411d9b8cfa417c158b770ee2cdf1cf: Status 404 returned error can't find the container with id bc66d97e073c8610d57e2e9b89438cf1dd411d9b8cfa417c158b770ee2cdf1cf Nov 02 20:13:52.886489 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:52.886447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:52.886707 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:52.886545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:13:52.886707 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:52.886682 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:13:52.886845 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:52.886748 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs podName:30d6b618-cc4e-48b8-a7b4-9649a4242aad nodeName:}" failed. No retries permitted until 2026-11-02 20:13:54.886728792 +0000 UTC m=+5.298864479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs") pod "network-metrics-daemon-5kzc7" (UID: "30d6b618-cc4e-48b8-a7b4-9649a4242aad") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:13:52.886845 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:52.886825 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 20:13:52.886987 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:52.886859 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret podName:afe07c4f-7282-4239-9761-c61ef4153cd9 nodeName:}" failed. No retries permitted until 2026-11-02 20:13:54.886847998 +0000 UTC m=+5.298983697 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret") pod "global-pull-secret-syncer-jjtbq" (UID: "afe07c4f-7282-4239-9761-c61ef4153cd9") : object "kube-system"/"original-pull-secret" not registered Nov 02 20:13:52.987424 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:52.987387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t22cd\" (UniqueName: \"kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd\") pod \"network-check-target-rphxh\" (UID: \"f8dd1189-9c04-4ba1-a349-04e7dabcab52\") " pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:13:52.987969 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:52.987548 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 20:13:52.987969 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:52.987574 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 20:13:52.987969 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:52.987588 2576 projected.go:196] Error preparing data for projected volume kube-api-access-t22cd for pod openshift-network-diagnostics/network-check-target-rphxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:13:52.987969 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:52.987649 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd podName:f8dd1189-9c04-4ba1-a349-04e7dabcab52 nodeName:}" failed. No retries permitted until 2026-11-02 20:13:54.987627349 +0000 UTC m=+5.399763037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t22cd" (UniqueName: "kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd") pod "network-check-target-rphxh" (UID: "f8dd1189-9c04-4ba1-a349-04e7dabcab52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:13:53.044294 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:53.044264 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:13:53.044464 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:53.044436 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kzc7" podUID="30d6b618-cc4e-48b8-a7b4-9649a4242aad" Nov 02 20:13:53.044536 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:53.044521 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:13:53.044620 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:53.044598 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rphxh" podUID="f8dd1189-9c04-4ba1-a349-04e7dabcab52" Nov 02 20:13:53.044691 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:53.044672 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:53.044769 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:53.044749 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjtbq" podUID="afe07c4f-7282-4239-9761-c61ef4153cd9" Nov 02 20:13:53.064625 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:53.064592 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7pl28" event={"ID":"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd","Type":"ContainerStarted","Data":"bc66d97e073c8610d57e2e9b89438cf1dd411d9b8cfa417c158b770ee2cdf1cf"} Nov 02 20:13:53.073135 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:53.073101 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-26nmv" event={"ID":"39e68d28-771c-48e8-ba38-f8aad80a6bf9","Type":"ContainerStarted","Data":"14e005f90d52a2dc6b1bcb210c65223609ecf4699f6910ad0cf71b52d3e38265"} Nov 02 20:13:53.284915 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:53.284794 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 20:08:51 +0000 UTC" deadline="2028-07-02 09:59:22.899379913 +0000 UTC" Nov 02 20:13:53.284915 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:53.284862 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14581h45m29.614522589s" Nov 02 20:13:54.903384 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:54.903298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:13:54.903384 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:54.903365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:54.904265 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:54.903484 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 20:13:54.904265 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:54.903547 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret podName:afe07c4f-7282-4239-9761-c61ef4153cd9 nodeName:}" failed. No retries permitted until 2026-11-02 20:13:58.903524714 +0000 UTC m=+9.315660417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret") pod "global-pull-secret-syncer-jjtbq" (UID: "afe07c4f-7282-4239-9761-c61ef4153cd9") : object "kube-system"/"original-pull-secret" not registered Nov 02 20:13:54.904265 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:54.903614 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:13:54.904265 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:54.903649 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs podName:30d6b618-cc4e-48b8-a7b4-9649a4242aad nodeName:}" failed. No retries permitted until 2026-11-02 20:13:58.90363696 +0000 UTC m=+9.315772662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs") pod "network-metrics-daemon-5kzc7" (UID: "30d6b618-cc4e-48b8-a7b4-9649a4242aad") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:13:55.004357 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:55.003911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t22cd\" (UniqueName: \"kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd\") pod \"network-check-target-rphxh\" (UID: \"f8dd1189-9c04-4ba1-a349-04e7dabcab52\") " pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:13:55.004357 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:55.004156 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 20:13:55.004357 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:55.004180 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 20:13:55.004357 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:55.004195 2576 projected.go:196] Error preparing data for projected volume kube-api-access-t22cd for pod openshift-network-diagnostics/network-check-target-rphxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:13:55.004357 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:55.004315 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd podName:f8dd1189-9c04-4ba1-a349-04e7dabcab52 nodeName:}" failed. No retries permitted until 2026-11-02 20:13:59.004235391 +0000 UTC m=+9.416371096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t22cd" (UniqueName: "kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd") pod "network-check-target-rphxh" (UID: "f8dd1189-9c04-4ba1-a349-04e7dabcab52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:13:55.044086 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:55.044016 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:55.044372 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:55.044184 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjtbq" podUID="afe07c4f-7282-4239-9761-c61ef4153cd9" Nov 02 20:13:55.044372 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:55.044249 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:13:55.044372 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:55.044335 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kzc7" podUID="30d6b618-cc4e-48b8-a7b4-9649a4242aad" Nov 02 20:13:55.044558 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:55.044381 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:13:55.044558 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:55.044447 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rphxh" podUID="f8dd1189-9c04-4ba1-a349-04e7dabcab52" Nov 02 20:13:57.044477 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:57.044250 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:57.044956 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:57.044299 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:13:57.044956 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:57.044594 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjtbq" podUID="afe07c4f-7282-4239-9761-c61ef4153cd9" Nov 02 20:13:57.044956 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:57.044670 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kzc7" podUID="30d6b618-cc4e-48b8-a7b4-9649a4242aad" Nov 02 20:13:57.044956 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:57.044329 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:13:57.044956 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:57.044781 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rphxh" podUID="f8dd1189-9c04-4ba1-a349-04e7dabcab52" Nov 02 20:13:57.125700 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:57.124707 2576 generic.go:356] "Generic (PLEG): container finished" podID="a138b9a8-1f14-42a3-97f6-9e2612f83470" containerID="ab0e35440e23a1ac0c11ed1aca20824d5ad11e2d44ee51eaff92fd9a93c9a6df" exitCode=0 Nov 02 20:13:57.125700 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:57.124810 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvfmm" event={"ID":"a138b9a8-1f14-42a3-97f6-9e2612f83470","Type":"ContainerDied","Data":"ab0e35440e23a1ac0c11ed1aca20824d5ad11e2d44ee51eaff92fd9a93c9a6df"} Nov 02 20:13:57.130916 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:57.130888 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jbk5q" event={"ID":"b51723af-6a33-4535-91c0-905ce2a51b24","Type":"ContainerStarted","Data":"83c48e6bb62326327ba66caf3d5d092ceb0113b30d2095ea295a1e55aae8d980"} Nov 02 20:13:57.134743 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:57.134717 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b86cdc381e2e2430d60e71836e38622" containerID="b2b37f6fb8b2176678cb69ccd70f080c5c5ad01d4eed8dc5c205924ce6adca37" exitCode=0 Nov 02 20:13:57.134842 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:57.134759 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" event={"ID":"4b86cdc381e2e2430d60e71836e38622","Type":"ContainerDied","Data":"b2b37f6fb8b2176678cb69ccd70f080c5c5ad01d4eed8dc5c205924ce6adca37"} Nov 02 20:13:57.560964 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:57.560896 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jbk5q" podStartSLOduration=2.3258751269999998 podStartE2EDuration="7.560874411s" totalImagesPullingTime="5.234999284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:57.560069319 +0000 UTC m=+7.972205025" watchObservedRunningTime="2026-11-02 20:13:57.560874411 +0000 UTC m=+7.973010117" Nov 02 20:13:58.141386 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:58.140745 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" event={"ID":"b918fb29af61c47ab1271b4e381400d1","Type":"ContainerStarted","Data":"01d201fb399e13b10ad3798372fda7bceb4995ddc731826c065c48d531abe0b8"} Nov 02 20:13:58.146287 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:58.146198 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" event={"ID":"4b86cdc381e2e2430d60e71836e38622","Type":"ContainerStarted","Data":"8b3b0eecee7ca50a29def95a73a68308d72a6da1803ab884e50396673c21d10a"} Nov 02 20:13:58.362057 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:58.361966 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" podStartSLOduration=8.361945856 podStartE2EDuration="8.361945856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:58.250834516 +0000 UTC m=+8.662970222" watchObservedRunningTime="2026-11-02 20:13:58.361945856 +0000 UTC m=+8.774081564" Nov 02 20:13:58.362239 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:58.362136 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" podStartSLOduration=8.362124972 podStartE2EDuration="8.362124972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:58.360998208 +0000 UTC m=+8.773133916" watchObservedRunningTime="2026-11-02 20:13:58.362124972 +0000 UTC m=+8.774260678" Nov 02 20:13:58.941655 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:58.941566 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:58.941655 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:58.941650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:13:58.941865 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:58.941760 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:13:58.941865 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:58.941812 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs podName:30d6b618-cc4e-48b8-a7b4-9649a4242aad nodeName:}" failed. No retries permitted until 2026-11-02 20:14:06.941795934 +0000 UTC m=+17.353931632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs") pod "network-metrics-daemon-5kzc7" (UID: "30d6b618-cc4e-48b8-a7b4-9649a4242aad") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:13:58.941976 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:58.941873 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 20:13:58.941976 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:58.941902 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret podName:afe07c4f-7282-4239-9761-c61ef4153cd9 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:06.941891583 +0000 UTC m=+17.354027267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret") pod "global-pull-secret-syncer-jjtbq" (UID: "afe07c4f-7282-4239-9761-c61ef4153cd9") : object "kube-system"/"original-pull-secret" not registered Nov 02 20:13:59.043024 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:59.042981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t22cd\" (UniqueName: \"kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd\") pod \"network-check-target-rphxh\" (UID: \"f8dd1189-9c04-4ba1-a349-04e7dabcab52\") " pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:13:59.043213 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:59.043195 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 20:13:59.043213 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:59.043214 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 20:13:59.043330 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:59.043226 2576 projected.go:196] Error preparing data for projected volume kube-api-access-t22cd for pod openshift-network-diagnostics/network-check-target-rphxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:13:59.043330 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:59.043279 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd podName:f8dd1189-9c04-4ba1-a349-04e7dabcab52 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:07.043261261 +0000 UTC m=+17.455396947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t22cd" (UniqueName: "kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd") pod "network-check-target-rphxh" (UID: "f8dd1189-9c04-4ba1-a349-04e7dabcab52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:13:59.043487 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:59.043465 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:13:59.043594 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:59.043571 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjtbq" podUID="afe07c4f-7282-4239-9761-c61ef4153cd9" Nov 02 20:13:59.043669 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:59.043658 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:13:59.043756 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:59.043739 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kzc7" podUID="30d6b618-cc4e-48b8-a7b4-9649a4242aad" Nov 02 20:13:59.043840 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:59.043793 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:13:59.043894 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:13:59.043858 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rphxh" podUID="f8dd1189-9c04-4ba1-a349-04e7dabcab52" Nov 02 20:13:59.154418 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:59.154349 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7pl28" event={"ID":"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd","Type":"ContainerStarted","Data":"964bacb8ff731bc4f7d559d2980cb6e78ef69489b13880d3a9eb0babfbd45858"} Nov 02 20:13:59.157820 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:59.157788 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" event={"ID":"1e96f673-1cf6-48f1-a55d-347662c75b16","Type":"ContainerStarted","Data":"15d3e7737211770dfd064fcdf0c37b22eeb95d442610f15c638edb9282e078d6"} Nov 02 20:13:59.903065 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:59.902417 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:13:59.903254 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:13:59.903070 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:14:00.015941 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:00.015886 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7pl28" podStartSLOduration=3.8922814470000002 podStartE2EDuration="10.015873062s" totalImagesPullingTime="6.123591615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:59.264669503 +0000 UTC m=+9.676805209" watchObservedRunningTime="2026-11-02 20:14:00.015873062 +0000 UTC m=+10.428008766" Nov 02 20:14:00.496910 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:00.496849 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 20:14:01.031012 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:01.030855 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T20:14:00.49686959Z","UUID":"05a6aef5-1c2a-4874-867c-2c7738eaa14e","Handler":null,"Name":"","Endpoint":""} Nov 02 20:14:01.032643 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:01.032623 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 20:14:01.032643 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:01.032653 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 20:14:01.043500 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:01.043476 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:01.043621 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:01.043600 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjtbq" podUID="afe07c4f-7282-4239-9761-c61ef4153cd9" Nov 02 20:14:01.043702 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:01.043659 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:01.043752 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:01.043722 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rphxh" podUID="f8dd1189-9c04-4ba1-a349-04e7dabcab52" Nov 02 20:14:01.043798 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:01.043767 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:01.043983 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:01.043957 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kzc7" podUID="30d6b618-cc4e-48b8-a7b4-9649a4242aad" Nov 02 20:14:01.164772 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:01.164735 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" event={"ID":"1e96f673-1cf6-48f1-a55d-347662c75b16","Type":"ContainerStarted","Data":"6d1b2030c30130ecb8010db6a2c0e529ee4f370c7f15ae0ef03523f57493d918"} Nov 02 20:14:01.166756 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:01.166726 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gxrvt" event={"ID":"9fec747e-0efa-40f5-afaf-d45d820700b1","Type":"ContainerStarted","Data":"8f180c4d953ba0b7121ae2fa7b3703f058849306f6d7fe1e067664b348c54fbd"} Nov 02 20:14:01.282366 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:01.282282 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-gxrvt" podStartSLOduration=2.324373235 podStartE2EDuration="11.282264047s" totalImagesPullingTime="8.957890812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:01.281797478 +0000 UTC m=+11.693933182" watchObservedRunningTime="2026-11-02 20:14:01.282264047 +0000 UTC m=+11.694399753" Nov 02 20:14:02.170851 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:02.170820 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-26nmv" event={"ID":"39e68d28-771c-48e8-ba38-f8aad80a6bf9","Type":"ContainerStarted","Data":"8daee4e5206f9202e3039161b89cd7169abe37bcb066fd38b7895d1d791a30a1"} Nov 02 20:14:02.172927 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:02.172899 2576 generic.go:356] "Generic (PLEG): container finished" podID="a138b9a8-1f14-42a3-97f6-9e2612f83470" containerID="dd9126013f33695c76a8fbb9a97d79bd2bb240618723e6c0a7ce4371cd63b1c9" exitCode=0 Nov 02 20:14:02.173020 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:02.173008 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvfmm" event={"ID":"a138b9a8-1f14-42a3-97f6-9e2612f83470","Type":"ContainerDied","Data":"dd9126013f33695c76a8fbb9a97d79bd2bb240618723e6c0a7ce4371cd63b1c9"} Nov 02 20:14:02.176838 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:02.176818 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l54nk" event={"ID":"68a10036-5c78-4b01-b06d-075a727a5d0b","Type":"ContainerStarted","Data":"dee582c8676f35f96b5b93e75e7353a2e4a891cf68a8f24f1bd50100bfd3d46a"} Nov 02 20:14:02.280656 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:02.280609 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-26nmv" podStartSLOduration=3.496897271 podStartE2EDuration="12.280594715s" totalImagesPullingTime="8.783697444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:02.280348128 +0000 UTC m=+12.692483834" watchObservedRunningTime="2026-11-02 20:14:02.280594715 +0000 UTC m=+12.692730420" Nov 02 20:14:02.507496 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:02.506655 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l54nk" podStartSLOduration=3.461555832 podStartE2EDuration="12.506636466s" totalImagesPullingTime="9.045080634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:02.389941131 +0000 UTC m=+12.802076836" watchObservedRunningTime="2026-11-02 20:14:02.506636466 +0000 UTC m=+12.918772172" Nov 02 20:14:03.044112 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:03.044014 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:03.044112 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:03.044041 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:03.044318 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:03.044195 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjtbq" podUID="afe07c4f-7282-4239-9761-c61ef4153cd9" Nov 02 20:14:03.044318 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:03.044244 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:03.044415 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:03.044347 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kzc7" podUID="30d6b618-cc4e-48b8-a7b4-9649a4242aad" Nov 02 20:14:03.044462 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:03.044425 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rphxh" podUID="f8dd1189-9c04-4ba1-a349-04e7dabcab52" Nov 02 20:14:03.179644 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:03.179607 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" event={"ID":"1e96f673-1cf6-48f1-a55d-347662c75b16","Type":"ContainerStarted","Data":"1fc0d80a60170ec3d0c7f7622558d68081a2767acb5f40a233fe6825979cb36a"} Nov 02 20:14:03.293718 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:03.293471 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" podStartSLOduration=1.677434512 podStartE2EDuration="12.293452337s" totalImagesPullingTime="10.616017825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:13:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:03.292736009 +0000 UTC m=+13.704871715" watchObservedRunningTime="2026-11-02 20:14:03.293452337 +0000 UTC m=+13.705588043" Nov 02 20:14:04.184328 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:04.184251 2576 generic.go:356] "Generic (PLEG): container finished" podID="a138b9a8-1f14-42a3-97f6-9e2612f83470" containerID="e2685dcaa9535d0d0ba80e466996feffc2b99d56d55aa2571f31c80e934409ee" exitCode=0 Nov 02 20:14:04.184908 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:04.184332 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvfmm" event={"ID":"a138b9a8-1f14-42a3-97f6-9e2612f83470","Type":"ContainerDied","Data":"e2685dcaa9535d0d0ba80e466996feffc2b99d56d55aa2571f31c80e934409ee"} Nov 02 20:14:04.185982 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:04.185951 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" event={"ID":"bcc543bf-8ffd-488d-999a-2b8da64e9c31","Type":"ContainerStarted","Data":"7f4e625bbb79daad4b31f937934a5512c95773d1ce68e0ba0e87de28ea5a3d7e"} Nov 02 20:14:04.438365 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:04.438251 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" podStartSLOduration=2.8831837030000003 podStartE2EDuration="14.438232959s" totalImagesPullingTime="11.555049256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:04.437957012 +0000 UTC m=+14.850092718" watchObservedRunningTime="2026-11-02 20:14:04.438232959 +0000 UTC m=+14.850368695" Nov 02 20:14:05.043731 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:05.043695 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:05.043906 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:05.043709 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:05.043906 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:05.043808 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rphxh" podUID="f8dd1189-9c04-4ba1-a349-04e7dabcab52" Nov 02 20:14:05.043906 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:05.043840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:05.044050 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:05.043919 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kzc7" podUID="30d6b618-cc4e-48b8-a7b4-9649a4242aad" Nov 02 20:14:05.044050 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:05.043994 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjtbq" podUID="afe07c4f-7282-4239-9761-c61ef4153cd9" Nov 02 20:14:06.192588 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:06.192553 2576 generic.go:356] "Generic (PLEG): container finished" podID="a138b9a8-1f14-42a3-97f6-9e2612f83470" containerID="54295e0ef15084eaf2104c3c6e4bede2665a7d0d94e514eff60adac8c5007cbb" exitCode=0 Nov 02 20:14:06.193431 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:06.192631 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvfmm" event={"ID":"a138b9a8-1f14-42a3-97f6-9e2612f83470","Type":"ContainerDied","Data":"54295e0ef15084eaf2104c3c6e4bede2665a7d0d94e514eff60adac8c5007cbb"} Nov 02 20:14:07.003949 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:07.003908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:07.004155 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:07.003969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:07.004155 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:07.004074 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:14:07.004155 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:07.004107 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 20:14:07.004316 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:07.004166 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs podName:30d6b618-cc4e-48b8-a7b4-9649a4242aad nodeName:}" failed. No retries permitted until 2026-11-02 20:14:23.004144982 +0000 UTC m=+33.416280688 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs") pod "network-metrics-daemon-5kzc7" (UID: "30d6b618-cc4e-48b8-a7b4-9649a4242aad") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:14:07.004316 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:07.004188 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret podName:afe07c4f-7282-4239-9761-c61ef4153cd9 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:23.004178214 +0000 UTC m=+33.416313897 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret") pod "global-pull-secret-syncer-jjtbq" (UID: "afe07c4f-7282-4239-9761-c61ef4153cd9") : object "kube-system"/"original-pull-secret" not registered Nov 02 20:14:07.044013 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:07.043989 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:07.044158 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:07.044065 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:07.044228 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:07.044167 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:07.044228 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:07.044169 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjtbq" podUID="afe07c4f-7282-4239-9761-c61ef4153cd9" Nov 02 20:14:07.044312 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:07.044250 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rphxh" podUID="f8dd1189-9c04-4ba1-a349-04e7dabcab52" Nov 02 20:14:07.044364 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:07.044311 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kzc7" podUID="30d6b618-cc4e-48b8-a7b4-9649a4242aad" Nov 02 20:14:07.104487 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:07.104454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t22cd\" (UniqueName: \"kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd\") pod \"network-check-target-rphxh\" (UID: \"f8dd1189-9c04-4ba1-a349-04e7dabcab52\") " pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:07.104615 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:07.104587 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 20:14:07.104615 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:07.104606 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 20:14:07.104706 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:07.104618 2576 projected.go:196] Error preparing data for projected volume kube-api-access-t22cd for pod openshift-network-diagnostics/network-check-target-rphxh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:14:07.104706 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:07.104673 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd podName:f8dd1189-9c04-4ba1-a349-04e7dabcab52 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:23.10465515 +0000 UTC m=+33.516790836 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t22cd" (UniqueName: "kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd") pod "network-check-target-rphxh" (UID: "f8dd1189-9c04-4ba1-a349-04e7dabcab52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:14:08.199385 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:08.199205 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" event={"ID":"64f527e7-dcc8-49be-8c15-a078a7aed186","Type":"ContainerStarted","Data":"79a28b8d9ad1da3b8f78cc79379fd19012d10bacb55e5c926ddecb0fc002a453"} Nov 02 20:14:08.199780 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:08.199430 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" event={"ID":"64f527e7-dcc8-49be-8c15-a078a7aed186","Type":"ContainerStarted","Data":"426de13c77d19d3f8a86c05606ade854b2d132d2ac2fd43be76d2f93013facbe"} Nov 02 20:14:08.199780 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:08.199467 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" event={"ID":"64f527e7-dcc8-49be-8c15-a078a7aed186","Type":"ContainerStarted","Data":"36c2088e75b87166a12faaf30d0fc762bdac2a69a9feddcaf5b8a5d3abd97bbe"} Nov 02 20:14:08.199780 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:08.199513 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" event={"ID":"64f527e7-dcc8-49be-8c15-a078a7aed186","Type":"ContainerStarted","Data":"1ca35fecdbda7f6668bc6e8bef5a8361d0f70703ab7456acb27b6eb019add83d"} Nov 02 20:14:08.199780 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:08.199548 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" event={"ID":"64f527e7-dcc8-49be-8c15-a078a7aed186","Type":"ContainerStarted","Data":"e28fbf12dbf911d25bd09ddda0ebbef5e6804e568c5fc13806c9bc2de0da2c64"} Nov 02 20:14:09.043528 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:09.043362 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:09.043653 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:09.043367 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:09.043699 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:09.043388 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:09.043730 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:09.043692 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kzc7" podUID="30d6b618-cc4e-48b8-a7b4-9649a4242aad" Nov 02 20:14:09.043785 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:09.043588 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rphxh" podUID="f8dd1189-9c04-4ba1-a349-04e7dabcab52" Nov 02 20:14:09.043785 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:09.043754 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjtbq" podUID="afe07c4f-7282-4239-9761-c61ef4153cd9" Nov 02 20:14:09.204100 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:09.204060 2576 generic.go:356] "Generic (PLEG): container finished" podID="a138b9a8-1f14-42a3-97f6-9e2612f83470" containerID="343e8c1f2d0e861a5f9a4a9deba0ac3befedf047a80020f8b3c75954636c5f03" exitCode=0 Nov 02 20:14:09.204520 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:09.204112 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvfmm" event={"ID":"a138b9a8-1f14-42a3-97f6-9e2612f83470","Type":"ContainerDied","Data":"343e8c1f2d0e861a5f9a4a9deba0ac3befedf047a80020f8b3c75954636c5f03"} Nov 02 20:14:09.207517 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:09.207492 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" event={"ID":"64f527e7-dcc8-49be-8c15-a078a7aed186","Type":"ContainerStarted","Data":"8275b6214a552bfa1fc7ca7b1c09a22893e8b0a868f80d0e82673ed0c1ea89a7"} Nov 02 20:14:10.212737 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:10.212702 2576 generic.go:356] "Generic (PLEG): container finished" podID="a138b9a8-1f14-42a3-97f6-9e2612f83470" containerID="67383d938a3ca80383ba6bc4a24da54c321cc3e4b4ede1ed449ec389de697968" exitCode=0 Nov 02 20:14:10.213182 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:10.212764 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvfmm" event={"ID":"a138b9a8-1f14-42a3-97f6-9e2612f83470","Type":"ContainerDied","Data":"67383d938a3ca80383ba6bc4a24da54c321cc3e4b4ede1ed449ec389de697968"} Nov 02 20:14:11.044205 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:11.044174 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:11.044205 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:11.044212 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:11.044409 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:11.044288 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kzc7" podUID="30d6b618-cc4e-48b8-a7b4-9649a4242aad" Nov 02 20:14:11.044409 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:11.044297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:11.044485 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:11.044402 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rphxh" podUID="f8dd1189-9c04-4ba1-a349-04e7dabcab52" Nov 02 20:14:11.044521 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:11.044481 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjtbq" podUID="afe07c4f-7282-4239-9761-c61ef4153cd9" Nov 02 20:14:11.219069 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:11.219009 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bvfmm" event={"ID":"a138b9a8-1f14-42a3-97f6-9e2612f83470","Type":"ContainerStarted","Data":"e92de95c2ceefa552ab9448bb391b7ba57d50da45cb0d11df7d5ab96a07262e6"} Nov 02 20:14:11.223827 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:11.223806 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" event={"ID":"64f527e7-dcc8-49be-8c15-a078a7aed186","Type":"ContainerStarted","Data":"ae1f26fdafda97558de9fec1559482592940572a3c56ed4d26f669fb28c2d240"} Nov 02 20:14:11.335961 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:11.335879 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bvfmm" podStartSLOduration=8.062239515 podStartE2EDuration="21.335865715s" totalImagesPullingTime="13.2736262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:13:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:11.335304032 +0000 UTC m=+21.747439737" watchObservedRunningTime="2026-11-02 20:14:11.335865715 +0000 UTC m=+21.748001420" Nov 02 20:14:13.044139 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:13.044108 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:13.044903 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:13.044228 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:13.044903 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:13.044242 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rphxh" podUID="f8dd1189-9c04-4ba1-a349-04e7dabcab52" Nov 02 20:14:13.044903 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:13.044257 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:13.044903 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:13.044322 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjtbq" podUID="afe07c4f-7282-4239-9761-c61ef4153cd9" Nov 02 20:14:13.044903 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:13.044408 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kzc7" podUID="30d6b618-cc4e-48b8-a7b4-9649a4242aad" Nov 02 20:14:13.232559 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:13.232530 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" event={"ID":"64f527e7-dcc8-49be-8c15-a078a7aed186","Type":"ContainerStarted","Data":"29fb288510a437fb716423a3b0bf92877d1def43ab074adb7b4d989fc888e037"} Nov 02 20:14:13.232838 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:13.232801 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:13.232950 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:13.232878 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:13.248262 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:13.248245 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:13.497553 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:13.497517 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" podStartSLOduration=7.227885633 podStartE2EDuration="23.497505535s" totalImagesPullingTime="16.269619902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:13.497172501 +0000 UTC m=+23.909308207" watchObservedRunningTime="2026-11-02 20:14:13.497505535 +0000 UTC m=+23.909641240" Nov 02 20:14:14.235214 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:14.235179 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:14.250710 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:14.250684 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:15.043311 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:15.043283 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:15.043459 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:15.043350 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:15.043521 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:15.043467 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjtbq" podUID="afe07c4f-7282-4239-9761-c61ef4153cd9" Nov 02 20:14:15.043521 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:15.043500 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:15.043636 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:15.043610 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kzc7" podUID="30d6b618-cc4e-48b8-a7b4-9649a4242aad" Nov 02 20:14:15.043759 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:15.043729 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rphxh" podUID="f8dd1189-9c04-4ba1-a349-04e7dabcab52" Nov 02 20:14:15.065515 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:15.065493 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jjtbq"] Nov 02 20:14:15.068371 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:15.068350 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rphxh"] Nov 02 20:14:15.071007 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:15.070989 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5kzc7"] Nov 02 20:14:15.236550 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:15.236524 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:15.236550 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:15.236536 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:15.236550 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:15.236541 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:15.236958 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:15.236634 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kzc7" podUID="30d6b618-cc4e-48b8-a7b4-9649a4242aad" Nov 02 20:14:15.236958 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:15.236686 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjtbq" podUID="afe07c4f-7282-4239-9761-c61ef4153cd9" Nov 02 20:14:15.236958 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:15.236764 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rphxh" podUID="f8dd1189-9c04-4ba1-a349-04e7dabcab52" Nov 02 20:14:17.044402 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:17.044228 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:17.044847 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:17.044234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:17.044847 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:17.044482 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjtbq" podUID="afe07c4f-7282-4239-9761-c61ef4153cd9" Nov 02 20:14:17.044847 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:17.044245 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:17.044847 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:17.044589 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kzc7" podUID="30d6b618-cc4e-48b8-a7b4-9649a4242aad" Nov 02 20:14:17.044847 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:17.044664 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rphxh" podUID="f8dd1189-9c04-4ba1-a349-04e7dabcab52" Nov 02 20:14:17.167042 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:17.167017 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:14:17.167202 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:17.167179 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 20:14:17.167612 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:17.167600 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:14:19.044191 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:19.044158 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:19.044585 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:19.044256 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:19.044585 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:19.044281 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:19.044585 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:19.044261 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kzc7" podUID="30d6b618-cc4e-48b8-a7b4-9649a4242aad" Nov 02 20:14:19.044585 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:19.044341 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rphxh" podUID="f8dd1189-9c04-4ba1-a349-04e7dabcab52" Nov 02 20:14:19.044585 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:19.044407 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jjtbq" podUID="afe07c4f-7282-4239-9761-c61ef4153cd9" Nov 02 20:14:20.316425 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.316392 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeReady" Nov 02 20:14:20.316810 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.316534 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 20:14:20.463552 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.463527 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5bk6w"] Nov 02 20:14:20.467475 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.467460 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r"] Nov 02 20:14:20.467598 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.467581 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5bk6w" Nov 02 20:14:20.471547 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.471528 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7fd7l"] Nov 02 20:14:20.471683 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.471650 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:20.475190 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.474987 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7shr8"] Nov 02 20:14:20.475328 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.475310 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7fd7l" Nov 02 20:14:20.478295 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.478278 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2kxv2"] Nov 02 20:14:20.478402 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.478387 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7shr8" Nov 02 20:14:20.481901 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.481885 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm"] Nov 02 20:14:20.482022 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.482007 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:20.485131 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.485114 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6578786844-fgcrv"] Nov 02 20:14:20.485222 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.485206 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:20.488066 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.488049 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-89j9l"] Nov 02 20:14:20.488175 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.488155 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.491000 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.490984 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-bgd5x"] Nov 02 20:14:20.491147 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.491128 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:20.493980 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.493964 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8lr5b"] Nov 02 20:14:20.494070 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.494057 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-bgd5x" Nov 02 20:14:20.496743 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.496725 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf"] Nov 02 20:14:20.496847 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.496833 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:20.499658 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.499644 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:20.511138 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.511116 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6"] Nov 02 20:14:20.517051 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.517033 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-v9wrk"] Nov 02 20:14:20.517241 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.517218 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:20.522003 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.520861 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 20:14:20.522003 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.520929 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 20:14:20.522003 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.521098 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:14:20.522003 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.521513 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hfjnv\"" Nov 02 20:14:20.522003 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.521575 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 20:14:20.522003 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.521690 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 20:14:20.522003 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.521691 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 20:14:20.522874 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.522856 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-f6f5bd44c-4tksg"] Nov 02 20:14:20.523021 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.523007 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.525851 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.525829 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-6tfzr"] Nov 02 20:14:20.525952 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.525934 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:20.529006 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.528968 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d646d5849-btrvm"] Nov 02 20:14:20.529380 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.529362 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tfzr" Nov 02 20:14:20.533394 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.533374 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-278nr"] Nov 02 20:14:20.534292 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.534239 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.536737 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.536276 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5bk6w"] Nov 02 20:14:20.536737 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.536322 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r"] Nov 02 20:14:20.536737 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.536348 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7fd7l"] Nov 02 20:14:20.536737 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.536368 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7shr8"] Nov 02 20:14:20.536737 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.536389 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-bgd5x"] Nov 02 20:14:20.536737 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.536429 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm"] Nov 02 20:14:20.536737 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.536537 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-278nr" Nov 02 20:14:20.541162 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.541146 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf"] Nov 02 20:14:20.579286 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.579261 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6"] Nov 02 20:14:20.581747 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.581727 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-v9wrk"] Nov 02 20:14:20.582698 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.582676 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-278nr"] Nov 02 20:14:20.583757 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.583729 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-6tfzr"] Nov 02 20:14:20.583949 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.583930 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2kxv2"] Nov 02 20:14:20.584803 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.584787 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6578786844-fgcrv"] Nov 02 20:14:20.590233 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.590214 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-89j9l"] Nov 02 20:14:20.597240 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.596609 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8lr5b"] Nov 02 20:14:20.611214 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-default-certificate\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:20.611304 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611220 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/896fb858-68d1-462c-8a00-2b2b6b5a01af-serving-cert\") pod \"service-ca-operator-544d749c8c-6qvvf\" (UID: \"896fb858-68d1-462c-8a00-2b2b6b5a01af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:20.611304 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611240 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:20.611304 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d23cf40e-53e9-406e-92f8-ce628b1baa68-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6\" (UID: \"d23cf40e-53e9-406e-92f8-ce628b1baa68\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:20.611304 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611288 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8r65\" (UniqueName: \"kubernetes.io/projected/d23cf40e-53e9-406e-92f8-ce628b1baa68-kube-api-access-d8r65\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6\" (UID: \"d23cf40e-53e9-406e-92f8-ce628b1baa68\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:20.611427 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611340 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:20.611427 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f65dc226-70a9-4fcf-ae3b-28559c2ff235-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.611481 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611435 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99e51c7f-420e-4709-b098-2ac9f0eaaedb-cert\") pod \"ingress-canary-278nr\" (UID: \"99e51c7f-420e-4709-b098-2ac9f0eaaedb\") " pod="openshift-ingress-canary/ingress-canary-278nr" Nov 02 20:14:20.611512 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-89j9l\" (UID: \"9d5e73f9-907c-40b1-a596-b079ded3abfb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:20.611544 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611524 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg6gq\" (UniqueName: \"kubernetes.io/projected/22cd77ec-a3f8-4e80-8ffb-def069d70e34-kube-api-access-qg6gq\") pod \"kas-connection-checker-685895c7c9-7fd7l\" (UID: \"22cd77ec-a3f8-4e80-8ffb-def069d70e34\") " pod="kube-system/kas-connection-checker-685895c7c9-7fd7l" Nov 02 20:14:20.611575 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611557 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f65dc226-70a9-4fcf-ae3b-28559c2ff235-snapshots\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.611603 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-bound-sa-token\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.611603 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmsvc\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-kube-api-access-kmsvc\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.611661 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a6fea55b-74c2-40b1-9952-cd767e448317-config\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:20.611695 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611682 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bskm\" (UniqueName: \"kubernetes.io/projected/f65dc226-70a9-4fcf-ae3b-28559c2ff235-kube-api-access-8bskm\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.612574 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-image-registry-private-configuration\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.612574 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611743 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-bound-sa-token\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.612574 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzbsq\" (UniqueName: \"kubernetes.io/projected/5013aa7f-b2fd-4df0-b5e3-3478bef510ee-kube-api-access-kzbsq\") pod \"kas-connection-checker-685895c7c9-7shr8\" (UID: \"5013aa7f-b2fd-4df0-b5e3-3478bef510ee\") " pod="kube-system/kas-connection-checker-685895c7c9-7shr8" Nov 02 20:14:20.612574 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611792 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-installation-pull-secrets\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.612574 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611835 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d23cf40e-53e9-406e-92f8-ce628b1baa68-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6\" (UID: \"d23cf40e-53e9-406e-92f8-ce628b1baa68\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:20.612574 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02eded99-edf7-41a0-8866-549252be5fe0-ca-trust-extracted\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.612574 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/283c8ed3-38ac-4a45-82d0-ec5bd9245376-config-volume\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:20.612574 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.612574 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611925 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:20.612574 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611943 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx5xx\" (UniqueName: \"kubernetes.io/projected/99e51c7f-420e-4709-b098-2ac9f0eaaedb-kube-api-access-cx5xx\") pod \"ingress-canary-278nr\" (UID: \"99e51c7f-420e-4709-b098-2ac9f0eaaedb\") " pod="openshift-ingress-canary/ingress-canary-278nr" Nov 02 20:14:20.612574 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611965 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2xgj\" (UniqueName: \"kubernetes.io/projected/a6fea55b-74c2-40b1-9952-cd767e448317-kube-api-access-f2xgj\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:20.612574 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.611990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f65dc226-70a9-4fcf-ae3b-28559c2ff235-service-ca-bundle\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.612574 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8mlg\" (UniqueName: \"kubernetes.io/projected/cb6295a6-f31e-48a3-b4df-7d872a961161-kube-api-access-v8mlg\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:20.612574 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/283c8ed3-38ac-4a45-82d0-ec5bd9245376-tmp-dir\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:20.612574 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mk86\" (UniqueName: \"kubernetes.io/projected/896fb858-68d1-462c-8a00-2b2b6b5a01af-kube-api-access-6mk86\") pod \"service-ca-operator-544d749c8c-6qvvf\" (UID: \"896fb858-68d1-462c-8a00-2b2b6b5a01af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:20.612974 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:20.612974 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:20.612974 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a6fea55b-74c2-40b1-9952-cd767e448317-serving-cert\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:20.612974 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/896fb858-68d1-462c-8a00-2b2b6b5a01af-config\") pod \"service-ca-operator-544d749c8c-6qvvf\" (UID: \"896fb858-68d1-462c-8a00-2b2b6b5a01af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:20.612974 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612242 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpqp8\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-kube-api-access-xpqp8\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.612974 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-installation-pull-secrets\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.612974 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612282 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9d5e73f9-907c-40b1-a596-b079ded3abfb-nginx-conf\") pod \"networking-console-plugin-86c478bb88-89j9l\" (UID: \"9d5e73f9-907c-40b1-a596-b079ded3abfb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:20.612974 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-stats-auth\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:20.612974 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-image-registry-private-configuration\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.612974 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fknt\" (UniqueName: \"kubernetes.io/projected/f22a4aac-bd75-41d2-b1c2-9293bb405389-kube-api-access-2fknt\") pod \"volume-data-source-validator-58c9f5f46d-bgd5x\" (UID: \"f22a4aac-bd75-41d2-b1c2-9293bb405389\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-bgd5x" Nov 02 20:14:20.612974 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6fea26b0-2cef-4683-a20f-620ef0880bfc-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:20.612974 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f65dc226-70a9-4fcf-ae3b-28559c2ff235-tmp\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.612974 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:20.612974 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfwzl\" (UniqueName: \"kubernetes.io/projected/aef015e0-b0a9-4c9f-8140-dd74d8265e64-kube-api-access-tfwzl\") pod \"kas-connection-checker-685895c7c9-5bk6w\" (UID: \"aef015e0-b0a9-4c9f-8140-dd74d8265e64\") " pod="kube-system/kas-connection-checker-685895c7c9-5bk6w" Nov 02 20:14:20.612974 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f65dc226-70a9-4fcf-ae3b-28559c2ff235-serving-cert\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.613398 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-registry-certificates\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.613398 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612562 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-certificates\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.613398 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612598 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-trusted-ca\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.613398 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612621 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngldn\" (UniqueName: \"kubernetes.io/projected/283c8ed3-38ac-4a45-82d0-ec5bd9245376-kube-api-access-ngldn\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:20.613398 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6tkr\" (UniqueName: \"kubernetes.io/projected/bfe84946-82e9-456a-b81f-af887093749b-kube-api-access-b6tkr\") pod \"network-check-source-6f78d59f99-6tfzr\" (UID: \"bfe84946-82e9-456a-b81f-af887093749b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tfzr" Nov 02 20:14:20.613398 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612657 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.613398 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-trusted-ca\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.613398 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll47d\" (UniqueName: \"kubernetes.io/projected/6fea26b0-2cef-4683-a20f-620ef0880bfc-kube-api-access-ll47d\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:20.613398 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpz59\" (UniqueName: \"kubernetes.io/projected/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-kube-api-access-gpz59\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:20.613398 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6fea55b-74c2-40b1-9952-cd767e448317-trusted-ca\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:20.613398 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.612810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cb81e57-c7dc-4447-959f-d5496ffee976-ca-trust-extracted\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.618422 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.618400 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 20:14:20.618697 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.618678 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 20:14:20.618859 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.618835 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 20:14:20.618859 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.618853 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 20:14:20.619041 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.618880 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 20:14:20.619041 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.618945 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 20:14:20.619041 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.618970 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:14:20.619238 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.619069 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 20:14:20.619238 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.618756 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 20:14:20.619238 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.618757 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 20:14:20.619367 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.619274 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:14:20.619367 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.618835 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 20:14:20.619367 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.618942 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 20:14:20.619509 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.619371 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 20:14:20.619509 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.619394 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 20:14:20.619509 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.619406 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:14:20.619509 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.619424 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 20:14:20.619509 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.619432 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 20:14:20.619509 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.619453 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 20:14:20.619509 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.619460 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 20:14:20.619509 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.619472 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 20:14:20.619509 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.619479 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 20:14:20.619509 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.619488 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:14:20.625883 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.625866 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 20:14:20.628301 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.628280 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 20:14:20.667360 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.667343 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 20:14:20.676418 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.676402 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 20:14:20.676505 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.676458 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 20:14:20.678151 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.678130 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 20:14:20.678527 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.678494 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 20:14:20.678527 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.678502 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 20:14:20.678527 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.678519 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 20:14:20.678527 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.678526 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 20:14:20.679299 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.679284 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d646d5849-btrvm"] Nov 02 20:14:20.680631 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.680617 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 20:14:20.680718 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.680659 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 20:14:20.680718 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.680679 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 20:14:20.680718 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.680702 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 20:14:20.680718 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.680704 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 20:14:20.680927 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.680728 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 20:14:20.680982 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.680938 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 20:14:20.680982 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.680951 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 20:14:20.681208 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.680992 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 20:14:20.681208 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.680992 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fjr76\"" Nov 02 20:14:20.681208 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.681012 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5v4kq\"" Nov 02 20:14:20.681208 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.681008 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-74k7h\"" Nov 02 20:14:20.681208 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.681025 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-djbmg\"" Nov 02 20:14:20.681208 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.681006 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gv6xl\"" Nov 02 20:14:20.681208 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.681011 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-xtqwc\"" Nov 02 20:14:20.681208 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.681219 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-82tkg\"" Nov 02 20:14:20.681208 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.681234 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-r6qc8\"" Nov 02 20:14:20.681569 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.681236 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-q2gdn\"" Nov 02 20:14:20.681569 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.681397 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ng8kx\"" Nov 02 20:14:20.681569 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.681436 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 20:14:20.682202 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.682186 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f6f5bd44c-4tksg"] Nov 02 20:14:20.713743 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.713698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-bound-sa-token\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.713743 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.713724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzbsq\" (UniqueName: \"kubernetes.io/projected/5013aa7f-b2fd-4df0-b5e3-3478bef510ee-kube-api-access-kzbsq\") pod \"kas-connection-checker-685895c7c9-7shr8\" (UID: \"5013aa7f-b2fd-4df0-b5e3-3478bef510ee\") " pod="kube-system/kas-connection-checker-685895c7c9-7shr8" Nov 02 20:14:20.713743 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.713741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-installation-pull-secrets\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.713889 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.713757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d23cf40e-53e9-406e-92f8-ce628b1baa68-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6\" (UID: \"d23cf40e-53e9-406e-92f8-ce628b1baa68\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:20.713889 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.713780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02eded99-edf7-41a0-8866-549252be5fe0-ca-trust-extracted\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.713889 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.713807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/283c8ed3-38ac-4a45-82d0-ec5bd9245376-config-volume\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:20.713889 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.713835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.714117 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.713908 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 20:14:20.714117 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.713921 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6578786844-fgcrv: secret "image-registry-tls" not found Nov 02 20:14:20.714117 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.713940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:20.714117 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.713972 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls podName:02eded99-edf7-41a0-8866-549252be5fe0 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:21.213953577 +0000 UTC m=+31.626089260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls") pod "image-registry-6578786844-fgcrv" (UID: "02eded99-edf7-41a0-8866-549252be5fe0") : secret "image-registry-tls" not found Nov 02 20:14:20.714117 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.714010 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 20:14:20.714117 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx5xx\" (UniqueName: \"kubernetes.io/projected/99e51c7f-420e-4709-b098-2ac9f0eaaedb-kube-api-access-cx5xx\") pod \"ingress-canary-278nr\" (UID: \"99e51c7f-420e-4709-b098-2ac9f0eaaedb\") " pod="openshift-ingress-canary/ingress-canary-278nr" Nov 02 20:14:20.714117 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714044 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2xgj\" (UniqueName: \"kubernetes.io/projected/a6fea55b-74c2-40b1-9952-cd767e448317-kube-api-access-f2xgj\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:20.714117 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.714063 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs podName:cb6295a6-f31e-48a3-b4df-7d872a961161 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:21.214043524 +0000 UTC m=+31.626179224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs") pod "router-default-f6f5bd44c-4tksg" (UID: "cb6295a6-f31e-48a3-b4df-7d872a961161") : secret "router-metrics-certs-default" not found Nov 02 20:14:20.714465 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714121 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f65dc226-70a9-4fcf-ae3b-28559c2ff235-service-ca-bundle\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.714465 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8mlg\" (UniqueName: \"kubernetes.io/projected/cb6295a6-f31e-48a3-b4df-7d872a961161-kube-api-access-v8mlg\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:20.714465 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714183 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/283c8ed3-38ac-4a45-82d0-ec5bd9245376-tmp-dir\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:20.714465 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mk86\" (UniqueName: \"kubernetes.io/projected/896fb858-68d1-462c-8a00-2b2b6b5a01af-kube-api-access-6mk86\") pod \"service-ca-operator-544d749c8c-6qvvf\" (UID: \"896fb858-68d1-462c-8a00-2b2b6b5a01af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:20.714465 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:20.714465 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:20.714465 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714256 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02eded99-edf7-41a0-8866-549252be5fe0-ca-trust-extracted\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.714465 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a6fea55b-74c2-40b1-9952-cd767e448317-serving-cert\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:20.714465 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/896fb858-68d1-462c-8a00-2b2b6b5a01af-config\") pod \"service-ca-operator-544d749c8c-6qvvf\" (UID: \"896fb858-68d1-462c-8a00-2b2b6b5a01af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:20.714465 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpqp8\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-kube-api-access-xpqp8\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.714465 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-installation-pull-secrets\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.714465 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9d5e73f9-907c-40b1-a596-b079ded3abfb-nginx-conf\") pod \"networking-console-plugin-86c478bb88-89j9l\" (UID: \"9d5e73f9-907c-40b1-a596-b079ded3abfb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:20.714465 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-stats-auth\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:20.714465 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-image-registry-private-configuration\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.714465 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fknt\" (UniqueName: \"kubernetes.io/projected/f22a4aac-bd75-41d2-b1c2-9293bb405389-kube-api-access-2fknt\") pod \"volume-data-source-validator-58c9f5f46d-bgd5x\" (UID: \"f22a4aac-bd75-41d2-b1c2-9293bb405389\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-bgd5x" Nov 02 20:14:20.714465 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6fea26b0-2cef-4683-a20f-620ef0880bfc-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:20.715101 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f65dc226-70a9-4fcf-ae3b-28559c2ff235-tmp\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.715101 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714500 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:20.715101 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwzl\" (UniqueName: \"kubernetes.io/projected/aef015e0-b0a9-4c9f-8140-dd74d8265e64-kube-api-access-tfwzl\") pod \"kas-connection-checker-685895c7c9-5bk6w\" (UID: \"aef015e0-b0a9-4c9f-8140-dd74d8265e64\") " pod="kube-system/kas-connection-checker-685895c7c9-5bk6w" Nov 02 20:14:20.715101 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f65dc226-70a9-4fcf-ae3b-28559c2ff235-serving-cert\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.715101 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-registry-certificates\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.715101 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-certificates\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.715101 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714664 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-trusted-ca\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.715101 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714666 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/283c8ed3-38ac-4a45-82d0-ec5bd9245376-tmp-dir\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:20.715101 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngldn\" (UniqueName: \"kubernetes.io/projected/283c8ed3-38ac-4a45-82d0-ec5bd9245376-kube-api-access-ngldn\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:20.715101 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714730 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f65dc226-70a9-4fcf-ae3b-28559c2ff235-service-ca-bundle\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.715101 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714804 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/283c8ed3-38ac-4a45-82d0-ec5bd9245376-config-volume\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:20.715101 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714950 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f65dc226-70a9-4fcf-ae3b-28559c2ff235-tmp\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.715101 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.714986 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tkr\" (UniqueName: \"kubernetes.io/projected/bfe84946-82e9-456a-b81f-af887093749b-kube-api-access-b6tkr\") pod \"network-check-source-6f78d59f99-6tfzr\" (UID: \"bfe84946-82e9-456a-b81f-af887093749b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tfzr" Nov 02 20:14:20.715101 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.715023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.715101 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.715057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-trusted-ca\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.715101 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.715103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll47d\" (UniqueName: \"kubernetes.io/projected/6fea26b0-2cef-4683-a20f-620ef0880bfc-kube-api-access-ll47d\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:20.715721 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.715129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpz59\" (UniqueName: \"kubernetes.io/projected/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-kube-api-access-gpz59\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:20.715721 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.715177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6fea55b-74c2-40b1-9952-cd767e448317-trusted-ca\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:20.715721 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.715216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cb81e57-c7dc-4447-959f-d5496ffee976-ca-trust-extracted\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.715721 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.715246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-default-certificate\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:20.715721 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.715252 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 20:14:20.715721 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.715273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/896fb858-68d1-462c-8a00-2b2b6b5a01af-serving-cert\") pod \"service-ca-operator-544d749c8c-6qvvf\" (UID: \"896fb858-68d1-462c-8a00-2b2b6b5a01af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:20.715721 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.715297 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls podName:8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed nodeName:}" failed. No retries permitted until 2026-11-02 20:14:21.215283381 +0000 UTC m=+31.627419071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-77d6r" (UID: "8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed") : secret "samples-operator-tls" not found Nov 02 20:14:20.715721 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.715662 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-registry-certificates\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.716255 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.715745 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 20:14:20.716255 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.715758 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d646d5849-btrvm: secret "image-registry-tls" not found Nov 02 20:14:20.716255 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.715815 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-trusted-ca\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.716255 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.716004 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls podName:9cb81e57-c7dc-4447-959f-d5496ffee976 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:21.215910941 +0000 UTC m=+31.628046626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls") pod "image-registry-5d646d5849-btrvm" (UID: "9cb81e57-c7dc-4447-959f-d5496ffee976") : secret "image-registry-tls" not found Nov 02 20:14:20.716255 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.716223 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:20.716542 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.716519 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:20.716651 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.716636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d23cf40e-53e9-406e-92f8-ce628b1baa68-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6\" (UID: \"d23cf40e-53e9-406e-92f8-ce628b1baa68\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:20.716765 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.716751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8r65\" (UniqueName: \"kubernetes.io/projected/d23cf40e-53e9-406e-92f8-ce628b1baa68-kube-api-access-d8r65\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6\" (UID: \"d23cf40e-53e9-406e-92f8-ce628b1baa68\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:20.716879 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.716866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:20.716973 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.716959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f65dc226-70a9-4fcf-ae3b-28559c2ff235-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.717072 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.717059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99e51c7f-420e-4709-b098-2ac9f0eaaedb-cert\") pod \"ingress-canary-278nr\" (UID: \"99e51c7f-420e-4709-b098-2ac9f0eaaedb\") " pod="openshift-ingress-canary/ingress-canary-278nr" Nov 02 20:14:20.717205 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.717188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-89j9l\" (UID: \"9d5e73f9-907c-40b1-a596-b079ded3abfb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:20.717313 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.717296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg6gq\" (UniqueName: \"kubernetes.io/projected/22cd77ec-a3f8-4e80-8ffb-def069d70e34-kube-api-access-qg6gq\") pod \"kas-connection-checker-685895c7c9-7fd7l\" (UID: \"22cd77ec-a3f8-4e80-8ffb-def069d70e34\") " pod="kube-system/kas-connection-checker-685895c7c9-7fd7l" Nov 02 20:14:20.717419 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.717402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f65dc226-70a9-4fcf-ae3b-28559c2ff235-snapshots\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.717516 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.717502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-bound-sa-token\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.717614 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.717600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmsvc\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-kube-api-access-kmsvc\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.717713 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.717700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a6fea55b-74c2-40b1-9952-cd767e448317-config\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:20.717814 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.717801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bskm\" (UniqueName: \"kubernetes.io/projected/f65dc226-70a9-4fcf-ae3b-28559c2ff235-kube-api-access-8bskm\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.717904 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.717888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-image-registry-private-configuration\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.717999 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.717978 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 20:14:20.718069 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.718057 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls podName:283c8ed3-38ac-4a45-82d0-ec5bd9245376 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:21.218043666 +0000 UTC m=+31.630179362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls") pod "dns-default-8lr5b" (UID: "283c8ed3-38ac-4a45-82d0-ec5bd9245376") : secret "dns-default-metrics-tls" not found Nov 02 20:14:20.718159 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.716644 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9d5e73f9-907c-40b1-a596-b079ded3abfb-nginx-conf\") pod \"networking-console-plugin-86c478bb88-89j9l\" (UID: \"9d5e73f9-907c-40b1-a596-b079ded3abfb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:20.718159 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.717904 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d23cf40e-53e9-406e-92f8-ce628b1baa68-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6\" (UID: \"d23cf40e-53e9-406e-92f8-ce628b1baa68\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:20.718261 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.718248 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle podName:cb6295a6-f31e-48a3-b4df-7d872a961161 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:21.218232411 +0000 UTC m=+31.630368306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle") pod "router-default-f6f5bd44c-4tksg" (UID: "cb6295a6-f31e-48a3-b4df-7d872a961161") : configmap references non-existent config key: service-ca.crt Nov 02 20:14:20.718318 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.716897 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/896fb858-68d1-462c-8a00-2b2b6b5a01af-config\") pod \"service-ca-operator-544d749c8c-6qvvf\" (UID: \"896fb858-68d1-462c-8a00-2b2b6b5a01af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:20.718848 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.718817 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a6fea55b-74c2-40b1-9952-cd767e448317-serving-cert\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:20.718981 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.718956 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cb81e57-c7dc-4447-959f-d5496ffee976-ca-trust-extracted\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.719060 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.719021 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-trusted-ca\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.719730 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.719706 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f65dc226-70a9-4fcf-ae3b-28559c2ff235-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.721540 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.719852 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f65dc226-70a9-4fcf-ae3b-28559c2ff235-snapshots\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.721540 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.717202 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-certificates\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.721540 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.720005 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 20:14:20.721540 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.720053 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert podName:9d5e73f9-907c-40b1-a596-b079ded3abfb nodeName:}" failed. No retries permitted until 2026-11-02 20:14:21.220036742 +0000 UTC m=+31.632172439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-89j9l" (UID: "9d5e73f9-907c-40b1-a596-b079ded3abfb") : secret "networking-console-plugin-cert" not found Nov 02 20:14:20.721540 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.720606 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-installation-pull-secrets\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.721540 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.720898 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a6fea55b-74c2-40b1-9952-cd767e448317-config\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:20.721540 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.721202 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f65dc226-70a9-4fcf-ae3b-28559c2ff235-serving-cert\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.721540 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.721320 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:20.721540 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:20.721382 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls podName:6fea26b0-2cef-4683-a20f-620ef0880bfc nodeName:}" failed. No retries permitted until 2026-11-02 20:14:21.221365212 +0000 UTC m=+31.633500899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ndzpm" (UID: "6fea26b0-2cef-4683-a20f-620ef0880bfc") : secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:20.721540 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.721398 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-installation-pull-secrets\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.722376 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.721638 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d23cf40e-53e9-406e-92f8-ce628b1baa68-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6\" (UID: \"d23cf40e-53e9-406e-92f8-ce628b1baa68\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:20.722376 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.721772 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6fea55b-74c2-40b1-9952-cd767e448317-trusted-ca\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:20.722376 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.721840 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/896fb858-68d1-462c-8a00-2b2b6b5a01af-serving-cert\") pod \"service-ca-operator-544d749c8c-6qvvf\" (UID: \"896fb858-68d1-462c-8a00-2b2b6b5a01af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:20.722376 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.721939 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-image-registry-private-configuration\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.722849 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.722828 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6fea26b0-2cef-4683-a20f-620ef0880bfc-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:20.722956 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.722933 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-stats-auth\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:20.723275 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.723257 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99e51c7f-420e-4709-b098-2ac9f0eaaedb-cert\") pod \"ingress-canary-278nr\" (UID: \"99e51c7f-420e-4709-b098-2ac9f0eaaedb\") " pod="openshift-ingress-canary/ingress-canary-278nr" Nov 02 20:14:20.723607 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.723583 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-default-certificate\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:20.724171 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.724152 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-image-registry-private-configuration\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.772870 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.772834 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-bound-sa-token\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.772870 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.772859 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2xgj\" (UniqueName: \"kubernetes.io/projected/a6fea55b-74c2-40b1-9952-cd767e448317-kube-api-access-f2xgj\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:20.773123 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.773069 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzbsq\" (UniqueName: \"kubernetes.io/projected/5013aa7f-b2fd-4df0-b5e3-3478bef510ee-kube-api-access-kzbsq\") pod \"kas-connection-checker-685895c7c9-7shr8\" (UID: \"5013aa7f-b2fd-4df0-b5e3-3478bef510ee\") " pod="kube-system/kas-connection-checker-685895c7c9-7shr8" Nov 02 20:14:20.782299 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.782273 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mk86\" (UniqueName: \"kubernetes.io/projected/896fb858-68d1-462c-8a00-2b2b6b5a01af-kube-api-access-6mk86\") pod \"service-ca-operator-544d749c8c-6qvvf\" (UID: \"896fb858-68d1-462c-8a00-2b2b6b5a01af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:20.782410 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.782319 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwzl\" (UniqueName: \"kubernetes.io/projected/aef015e0-b0a9-4c9f-8140-dd74d8265e64-kube-api-access-tfwzl\") pod \"kas-connection-checker-685895c7c9-5bk6w\" (UID: \"aef015e0-b0a9-4c9f-8140-dd74d8265e64\") " pod="kube-system/kas-connection-checker-685895c7c9-5bk6w" Nov 02 20:14:20.782568 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.782547 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx5xx\" (UniqueName: \"kubernetes.io/projected/99e51c7f-420e-4709-b098-2ac9f0eaaedb-kube-api-access-cx5xx\") pod \"ingress-canary-278nr\" (UID: \"99e51c7f-420e-4709-b098-2ac9f0eaaedb\") " pod="openshift-ingress-canary/ingress-canary-278nr" Nov 02 20:14:20.782675 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.782644 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngldn\" (UniqueName: \"kubernetes.io/projected/283c8ed3-38ac-4a45-82d0-ec5bd9245376-kube-api-access-ngldn\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:20.783714 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.783687 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fknt\" (UniqueName: \"kubernetes.io/projected/f22a4aac-bd75-41d2-b1c2-9293bb405389-kube-api-access-2fknt\") pod \"volume-data-source-validator-58c9f5f46d-bgd5x\" (UID: \"f22a4aac-bd75-41d2-b1c2-9293bb405389\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-bgd5x" Nov 02 20:14:20.784032 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.784007 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8mlg\" (UniqueName: \"kubernetes.io/projected/cb6295a6-f31e-48a3-b4df-7d872a961161-kube-api-access-v8mlg\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:20.784377 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.784267 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpqp8\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-kube-api-access-xpqp8\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.784641 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.784598 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpz59\" (UniqueName: \"kubernetes.io/projected/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-kube-api-access-gpz59\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:20.785256 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.785232 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-bound-sa-token\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:20.785345 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.785295 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8r65\" (UniqueName: \"kubernetes.io/projected/d23cf40e-53e9-406e-92f8-ce628b1baa68-kube-api-access-d8r65\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6\" (UID: \"d23cf40e-53e9-406e-92f8-ce628b1baa68\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:20.787356 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.787333 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bskm\" (UniqueName: \"kubernetes.io/projected/f65dc226-70a9-4fcf-ae3b-28559c2ff235-kube-api-access-8bskm\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.787450 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.787349 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tkr\" (UniqueName: \"kubernetes.io/projected/bfe84946-82e9-456a-b81f-af887093749b-kube-api-access-b6tkr\") pod \"network-check-source-6f78d59f99-6tfzr\" (UID: \"bfe84946-82e9-456a-b81f-af887093749b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tfzr" Nov 02 20:14:20.787450 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.787338 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg6gq\" (UniqueName: \"kubernetes.io/projected/22cd77ec-a3f8-4e80-8ffb-def069d70e34-kube-api-access-qg6gq\") pod \"kas-connection-checker-685895c7c9-7fd7l\" (UID: \"22cd77ec-a3f8-4e80-8ffb-def069d70e34\") " pod="kube-system/kas-connection-checker-685895c7c9-7fd7l" Nov 02 20:14:20.787450 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.787394 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmsvc\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-kube-api-access-kmsvc\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:20.788553 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.788534 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll47d\" (UniqueName: \"kubernetes.io/projected/6fea26b0-2cef-4683-a20f-620ef0880bfc-kube-api-access-ll47d\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:20.792533 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.792514 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7fd7l" Nov 02 20:14:20.798900 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.798874 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7shr8" Nov 02 20:14:20.804663 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.804645 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:20.843662 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.843632 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-bgd5x" Nov 02 20:14:20.847619 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.847130 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:20.853348 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.853229 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:20.859269 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.859145 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:20.871187 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.871160 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tfzr" Nov 02 20:14:20.890694 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:20.883192 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-278nr" Nov 02 20:14:21.037321 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.037283 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:21.044264 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.044243 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:21.044373 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.044276 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:21.044427 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.044397 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:21.081328 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.081306 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5bk6w" Nov 02 20:14:21.094398 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.094336 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 20:14:21.094529 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.094509 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 20:14:21.094529 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.094513 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4wc8w\"" Nov 02 20:14:21.094632 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.094528 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7bq62\"" Nov 02 20:14:21.094632 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.094570 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s29pz\"" Nov 02 20:14:21.182789 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.182767 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7shr8"] Nov 02 20:14:21.185474 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:14:21.185447 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5013aa7f_b2fd_4df0_b5e3_3478bef510ee.slice/crio-ca238d1beee6588afe58ce0417b419e77f5fc5f70fda1e92edaef7c4d552b662 WatchSource:0}: Error finding container ca238d1beee6588afe58ce0417b419e77f5fc5f70fda1e92edaef7c4d552b662: Status 404 returned error can't find the container with id ca238d1beee6588afe58ce0417b419e77f5fc5f70fda1e92edaef7c4d552b662 Nov 02 20:14:21.189237 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.189199 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2kxv2"] Nov 02 20:14:21.191131 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:14:21.191114 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6fea55b_74c2_40b1_9952_cd767e448317.slice/crio-f34649d86715069ceade5514d1e1fc853e5db5d7b431b9df51b5d232244de298 WatchSource:0}: Error finding container f34649d86715069ceade5514d1e1fc853e5db5d7b431b9df51b5d232244de298: Status 404 returned error can't find the container with id f34649d86715069ceade5514d1e1fc853e5db5d7b431b9df51b5d232244de298 Nov 02 20:14:21.224490 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.224431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:21.224586 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.224493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:21.224586 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.224524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-89j9l\" (UID: \"9d5e73f9-907c-40b1-a596-b079ded3abfb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:21.224683 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.224596 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:21.224683 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.224624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:21.224683 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.224663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:21.224821 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.224691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:21.224821 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.224764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:21.224911 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.224894 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 20:14:21.224911 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.224908 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d646d5849-btrvm: secret "image-registry-tls" not found Nov 02 20:14:21.224996 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.224957 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls podName:9cb81e57-c7dc-4447-959f-d5496ffee976 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:22.224938867 +0000 UTC m=+32.637074568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls") pod "image-registry-5d646d5849-btrvm" (UID: "9cb81e57-c7dc-4447-959f-d5496ffee976") : secret "image-registry-tls" not found Nov 02 20:14:21.225058 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.225023 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:21.225135 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.225061 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls podName:6fea26b0-2cef-4683-a20f-620ef0880bfc nodeName:}" failed. No retries permitted until 2026-11-02 20:14:22.225049564 +0000 UTC m=+32.637185262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ndzpm" (UID: "6fea26b0-2cef-4683-a20f-620ef0880bfc") : secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:21.225195 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.225154 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle podName:cb6295a6-f31e-48a3-b4df-7d872a961161 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:22.225144428 +0000 UTC m=+32.637280125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle") pod "router-default-f6f5bd44c-4tksg" (UID: "cb6295a6-f31e-48a3-b4df-7d872a961161") : configmap references non-existent config key: service-ca.crt Nov 02 20:14:21.225249 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.225215 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 20:14:21.225296 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.225248 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert podName:9d5e73f9-907c-40b1-a596-b079ded3abfb nodeName:}" failed. No retries permitted until 2026-11-02 20:14:22.225237247 +0000 UTC m=+32.637372944 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-89j9l" (UID: "9d5e73f9-907c-40b1-a596-b079ded3abfb") : secret "networking-console-plugin-cert" not found Nov 02 20:14:21.225685 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.225584 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 20:14:21.225685 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.225601 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6578786844-fgcrv: secret "image-registry-tls" not found Nov 02 20:14:21.225685 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.225634 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls podName:02eded99-edf7-41a0-8866-549252be5fe0 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:22.225621826 +0000 UTC m=+32.637757523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls") pod "image-registry-6578786844-fgcrv" (UID: "02eded99-edf7-41a0-8866-549252be5fe0") : secret "image-registry-tls" not found Nov 02 20:14:21.225685 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.225685 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 20:14:21.225898 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.225714 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs podName:cb6295a6-f31e-48a3-b4df-7d872a961161 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:22.225704203 +0000 UTC m=+32.637839902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs") pod "router-default-f6f5bd44c-4tksg" (UID: "cb6295a6-f31e-48a3-b4df-7d872a961161") : secret "router-metrics-certs-default" not found Nov 02 20:14:21.225898 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.225770 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 20:14:21.225898 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.225802 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls podName:8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed nodeName:}" failed. No retries permitted until 2026-11-02 20:14:22.225792159 +0000 UTC m=+32.637927856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-77d6r" (UID: "8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed") : secret "samples-operator-tls" not found Nov 02 20:14:21.225898 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.225859 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 20:14:21.225898 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:21.225889 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls podName:283c8ed3-38ac-4a45-82d0-ec5bd9245376 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:22.225879693 +0000 UTC m=+32.638015379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls") pod "dns-default-8lr5b" (UID: "283c8ed3-38ac-4a45-82d0-ec5bd9245376") : secret "dns-default-metrics-tls" not found Nov 02 20:14:21.251550 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.251526 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" event={"ID":"a6fea55b-74c2-40b1-9952-cd767e448317","Type":"ContainerStarted","Data":"f34649d86715069ceade5514d1e1fc853e5db5d7b431b9df51b5d232244de298"} Nov 02 20:14:21.252731 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.252644 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7shr8" event={"ID":"5013aa7f-b2fd-4df0-b5e3-3478bef510ee","Type":"ContainerStarted","Data":"ca238d1beee6588afe58ce0417b419e77f5fc5f70fda1e92edaef7c4d552b662"} Nov 02 20:14:21.277352 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.277326 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf"] Nov 02 20:14:21.279651 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.279623 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-bgd5x"] Nov 02 20:14:21.280882 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:14:21.280862 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896fb858_68d1_462c_8a00_2b2b6b5a01af.slice/crio-af81f493cee046ae85b7be06bade3ce4506994eba8b5dda4076bf6413296b13b WatchSource:0}: Error finding container af81f493cee046ae85b7be06bade3ce4506994eba8b5dda4076bf6413296b13b: Status 404 returned error can't find the container with id af81f493cee046ae85b7be06bade3ce4506994eba8b5dda4076bf6413296b13b Nov 02 20:14:21.281530 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:14:21.281511 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22a4aac_bd75_41d2_b1c2_9293bb405389.slice/crio-ee3b20c44102840b9bf245e66c89b2e0941ff66d2bcd0140c2f3d052154e3f71 WatchSource:0}: Error finding container ee3b20c44102840b9bf245e66c89b2e0941ff66d2bcd0140c2f3d052154e3f71: Status 404 returned error can't find the container with id ee3b20c44102840b9bf245e66c89b2e0941ff66d2bcd0140c2f3d052154e3f71 Nov 02 20:14:21.287452 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.287430 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6"] Nov 02 20:14:21.290129 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:14:21.290101 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23cf40e_53e9_406e_92f8_ce628b1baa68.slice/crio-aa2af273ee991d426f21609d08d24b63d88a5cd6378432b8dc7f8ff622fd7937 WatchSource:0}: Error finding container aa2af273ee991d426f21609d08d24b63d88a5cd6378432b8dc7f8ff622fd7937: Status 404 returned error can't find the container with id aa2af273ee991d426f21609d08d24b63d88a5cd6378432b8dc7f8ff622fd7937 Nov 02 20:14:21.304530 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.304512 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7fd7l"] Nov 02 20:14:21.306262 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:14:21.306242 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cd77ec_a3f8_4e80_8ffb_def069d70e34.slice/crio-71c7ced32f7d40df0fdd5e71f6234eda39d1c2a35f033827b6472f75ccd11856 WatchSource:0}: Error finding container 71c7ced32f7d40df0fdd5e71f6234eda39d1c2a35f033827b6472f75ccd11856: Status 404 returned error can't find the container with id 71c7ced32f7d40df0fdd5e71f6234eda39d1c2a35f033827b6472f75ccd11856 Nov 02 20:14:21.308149 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.308130 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-6tfzr"] Nov 02 20:14:21.311141 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:14:21.311121 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe84946_82e9_456a_b81f_af887093749b.slice/crio-82c2cb92ed201c94477cd9681fb845f1e683c68a36351893b5b706f136e061c1 WatchSource:0}: Error finding container 82c2cb92ed201c94477cd9681fb845f1e683c68a36351893b5b706f136e061c1: Status 404 returned error can't find the container with id 82c2cb92ed201c94477cd9681fb845f1e683c68a36351893b5b706f136e061c1 Nov 02 20:14:21.311661 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.311645 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-v9wrk"] Nov 02 20:14:21.315319 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:14:21.315288 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65dc226_70a9_4fcf_ae3b_28559c2ff235.slice/crio-7873bc74331e14113791f9d5c98f8803f2300f77f1296ca2975e178b121fb289 WatchSource:0}: Error finding container 7873bc74331e14113791f9d5c98f8803f2300f77f1296ca2975e178b121fb289: Status 404 returned error can't find the container with id 7873bc74331e14113791f9d5c98f8803f2300f77f1296ca2975e178b121fb289 Nov 02 20:14:21.334876 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.334856 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-278nr"] Nov 02 20:14:21.337830 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:14:21.337221 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e51c7f_420e_4709_b098_2ac9f0eaaedb.slice/crio-bd0d3d1b1070f9fd7a896f2b22158a6cc02853c2bc49691f7dbd0c1fda88b9cc WatchSource:0}: Error finding container bd0d3d1b1070f9fd7a896f2b22158a6cc02853c2bc49691f7dbd0c1fda88b9cc: Status 404 returned error can't find the container with id bd0d3d1b1070f9fd7a896f2b22158a6cc02853c2bc49691f7dbd0c1fda88b9cc Nov 02 20:14:21.350356 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.350263 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:21.439982 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:21.439959 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5bk6w"] Nov 02 20:14:21.442907 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:14:21.442883 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef015e0_b0a9_4c9f_8140_dd74d8265e64.slice/crio-9f614f8ecc585a5a2ded495c5f6f82ce305420b1a56eba73a519e203e8c03310 WatchSource:0}: Error finding container 9f614f8ecc585a5a2ded495c5f6f82ce305420b1a56eba73a519e203e8c03310: Status 404 returned error can't find the container with id 9f614f8ecc585a5a2ded495c5f6f82ce305420b1a56eba73a519e203e8c03310 Nov 02 20:14:22.203998 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.203964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:22.235886 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.235857 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:22.236008 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.235899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:22.236008 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.235919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-89j9l\" (UID: \"9d5e73f9-907c-40b1-a596-b079ded3abfb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:22.236008 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.235963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:22.236191 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.236011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:22.236191 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236074 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:22.236191 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236132 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle podName:cb6295a6-f31e-48a3-b4df-7d872a961161 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:24.236111945 +0000 UTC m=+34.648247647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle") pod "router-default-f6f5bd44c-4tksg" (UID: "cb6295a6-f31e-48a3-b4df-7d872a961161") : configmap references non-existent config key: service-ca.crt Nov 02 20:14:22.236191 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236141 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 20:14:22.236191 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236167 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls podName:6fea26b0-2cef-4683-a20f-620ef0880bfc nodeName:}" failed. No retries permitted until 2026-11-02 20:14:24.236143303 +0000 UTC m=+34.648278989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ndzpm" (UID: "6fea26b0-2cef-4683-a20f-620ef0880bfc") : secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:22.236191 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.236061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:22.236427 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236204 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 20:14:22.236427 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236211 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 20:14:22.236427 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236229 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6578786844-fgcrv: secret "image-registry-tls" not found Nov 02 20:14:22.236427 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236229 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 20:14:22.236427 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236193 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert podName:9d5e73f9-907c-40b1-a596-b079ded3abfb nodeName:}" failed. No retries permitted until 2026-11-02 20:14:24.236177749 +0000 UTC m=+34.648313445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-89j9l" (UID: "9d5e73f9-907c-40b1-a596-b079ded3abfb") : secret "networking-console-plugin-cert" not found Nov 02 20:14:22.236427 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236283 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls podName:8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed nodeName:}" failed. No retries permitted until 2026-11-02 20:14:24.236252655 +0000 UTC m=+34.648388337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-77d6r" (UID: "8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed") : secret "samples-operator-tls" not found Nov 02 20:14:22.236427 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.236301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:22.236427 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236337 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs podName:cb6295a6-f31e-48a3-b4df-7d872a961161 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:24.236331786 +0000 UTC m=+34.648467469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs") pod "router-default-f6f5bd44c-4tksg" (UID: "cb6295a6-f31e-48a3-b4df-7d872a961161") : secret "router-metrics-certs-default" not found Nov 02 20:14:22.236427 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236356 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls podName:02eded99-edf7-41a0-8866-549252be5fe0 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:24.236349538 +0000 UTC m=+34.648485221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls") pod "image-registry-6578786844-fgcrv" (UID: "02eded99-edf7-41a0-8866-549252be5fe0") : secret "image-registry-tls" not found Nov 02 20:14:22.236427 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236393 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 20:14:22.236427 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.236431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:22.236800 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236491 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 20:14:22.236800 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236503 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d646d5849-btrvm: secret "image-registry-tls" not found Nov 02 20:14:22.236800 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236504 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls podName:283c8ed3-38ac-4a45-82d0-ec5bd9245376 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:24.236497776 +0000 UTC m=+34.648633458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls") pod "dns-default-8lr5b" (UID: "283c8ed3-38ac-4a45-82d0-ec5bd9245376") : secret "dns-default-metrics-tls" not found Nov 02 20:14:22.236800 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:22.236545 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls podName:9cb81e57-c7dc-4447-959f-d5496ffee976 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:24.236532285 +0000 UTC m=+34.648668000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls") pod "image-registry-5d646d5849-btrvm" (UID: "9cb81e57-c7dc-4447-959f-d5496ffee976") : secret "image-registry-tls" not found Nov 02 20:14:22.257095 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.257054 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7fd7l" event={"ID":"22cd77ec-a3f8-4e80-8ffb-def069d70e34","Type":"ContainerStarted","Data":"4ac4d3383539e0ed3cc486ade6492ed4f431e4a04a666b743058f939aaae5f11"} Nov 02 20:14:22.257194 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.257132 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7fd7l" event={"ID":"22cd77ec-a3f8-4e80-8ffb-def069d70e34","Type":"ContainerStarted","Data":"71c7ced32f7d40df0fdd5e71f6234eda39d1c2a35f033827b6472f75ccd11856"} Nov 02 20:14:22.259054 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.259032 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5bk6w" event={"ID":"aef015e0-b0a9-4c9f-8140-dd74d8265e64","Type":"ContainerStarted","Data":"6c16cc1832fa093b48d5c0c5fc5e94ef7e681074f62094f5f505e2404d6efea8"} Nov 02 20:14:22.259171 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.259074 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5bk6w" event={"ID":"aef015e0-b0a9-4c9f-8140-dd74d8265e64","Type":"ContainerStarted","Data":"9f614f8ecc585a5a2ded495c5f6f82ce305420b1a56eba73a519e203e8c03310"} Nov 02 20:14:22.260178 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.260151 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tfzr" event={"ID":"bfe84946-82e9-456a-b81f-af887093749b","Type":"ContainerStarted","Data":"82c2cb92ed201c94477cd9681fb845f1e683c68a36351893b5b706f136e061c1"} Nov 02 20:14:22.261514 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.261489 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-v9wrk" event={"ID":"f65dc226-70a9-4fcf-ae3b-28559c2ff235","Type":"ContainerStarted","Data":"7873bc74331e14113791f9d5c98f8803f2300f77f1296ca2975e178b121fb289"} Nov 02 20:14:22.262804 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.262755 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-278nr" event={"ID":"99e51c7f-420e-4709-b098-2ac9f0eaaedb","Type":"ContainerStarted","Data":"bd0d3d1b1070f9fd7a896f2b22158a6cc02853c2bc49691f7dbd0c1fda88b9cc"} Nov 02 20:14:22.264013 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.263992 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" event={"ID":"d23cf40e-53e9-406e-92f8-ce628b1baa68","Type":"ContainerStarted","Data":"aa2af273ee991d426f21609d08d24b63d88a5cd6378432b8dc7f8ff622fd7937"} Nov 02 20:14:22.265381 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.265357 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" event={"ID":"896fb858-68d1-462c-8a00-2b2b6b5a01af","Type":"ContainerStarted","Data":"af81f493cee046ae85b7be06bade3ce4506994eba8b5dda4076bf6413296b13b"} Nov 02 20:14:22.266824 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.266792 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-bgd5x" event={"ID":"f22a4aac-bd75-41d2-b1c2-9293bb405389","Type":"ContainerStarted","Data":"ee3b20c44102840b9bf245e66c89b2e0941ff66d2bcd0140c2f3d052154e3f71"} Nov 02 20:14:22.268556 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.268531 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7shr8" event={"ID":"5013aa7f-b2fd-4df0-b5e3-3478bef510ee","Type":"ContainerStarted","Data":"8b8570016049fd0a0eceef05c3d345c5ddee0db930787d77ced72a8e2b535356"} Nov 02 20:14:22.500373 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.500300 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:22.537745 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.537688 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5bk6w" podStartSLOduration=395.53767106 podStartE2EDuration="6m35.53767106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:22.536376628 +0000 UTC m=+32.948512335" watchObservedRunningTime="2026-11-02 20:14:22.53767106 +0000 UTC m=+32.949806766" Nov 02 20:14:22.647698 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.647633 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7shr8" podStartSLOduration=395.6476152 podStartE2EDuration="6m35.6476152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:22.647057188 +0000 UTC m=+33.059192897" watchObservedRunningTime="2026-11-02 20:14:22.6476152 +0000 UTC m=+33.059750948" Nov 02 20:14:22.760902 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:22.760801 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7fd7l" podStartSLOduration=395.760789899 podStartE2EDuration="6m35.760789899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:22.75943095 +0000 UTC m=+33.171566656" watchObservedRunningTime="2026-11-02 20:14:22.760789899 +0000 UTC m=+33.172925642" Nov 02 20:14:23.046058 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:23.045974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:23.046245 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:23.046152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:23.046331 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:23.046314 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 20:14:23.046453 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:23.046417 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs podName:30d6b618-cc4e-48b8-a7b4-9649a4242aad nodeName:}" failed. No retries permitted until 2026-11-02 20:14:55.046397294 +0000 UTC m=+65.458532996 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs") pod "network-metrics-daemon-5kzc7" (UID: "30d6b618-cc4e-48b8-a7b4-9649a4242aad") : secret "metrics-daemon-secret" not found Nov 02 20:14:23.049652 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:23.049616 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:23.147501 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:23.147464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t22cd\" (UniqueName: \"kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd\") pod \"network-check-target-rphxh\" (UID: \"f8dd1189-9c04-4ba1-a349-04e7dabcab52\") " pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:23.150812 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:23.150782 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t22cd\" (UniqueName: \"kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd\") pod \"network-check-target-rphxh\" (UID: \"f8dd1189-9c04-4ba1-a349-04e7dabcab52\") " pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:23.161487 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:23.161466 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:23.168666 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:23.168631 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:23.561202 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:23.561175 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rphxh"] Nov 02 20:14:23.563954 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:14:23.563924 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8dd1189_9c04_4ba1_a349_04e7dabcab52.slice/crio-b6cc565920073e6b826eb68c89265880f87092b390ddbfaf0f49c582b108cb60 WatchSource:0}: Error finding container b6cc565920073e6b826eb68c89265880f87092b390ddbfaf0f49c582b108cb60: Status 404 returned error can't find the container with id b6cc565920073e6b826eb68c89265880f87092b390ddbfaf0f49c582b108cb60 Nov 02 20:14:23.584051 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:23.584024 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jjtbq"] Nov 02 20:14:23.586448 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:14:23.586414 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe07c4f_7282_4239_9761_c61ef4153cd9.slice/crio-fbcf8209bfc27d3f3e1b23824d5dd3abfe29021d719ae75cb9fa223d7f3297dd WatchSource:0}: Error finding container fbcf8209bfc27d3f3e1b23824d5dd3abfe29021d719ae75cb9fa223d7f3297dd: Status 404 returned error can't find the container with id fbcf8209bfc27d3f3e1b23824d5dd3abfe29021d719ae75cb9fa223d7f3297dd Nov 02 20:14:23.674432 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:23.674402 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:23.963922 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:23.963888 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:24.261363 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:24.261265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:24.261363 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:24.261334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:24.261583 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:24.261370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:24.261583 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:24.261400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-89j9l\" (UID: \"9d5e73f9-907c-40b1-a596-b079ded3abfb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:24.261583 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:24.261454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:24.261583 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:24.261482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:24.261583 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:24.261522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:24.261583 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:24.261547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:24.261883 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.261697 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 20:14:24.261883 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.261759 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls podName:283c8ed3-38ac-4a45-82d0-ec5bd9245376 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:28.261740493 +0000 UTC m=+38.673876202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls") pod "dns-default-8lr5b" (UID: "283c8ed3-38ac-4a45-82d0-ec5bd9245376") : secret "dns-default-metrics-tls" not found Nov 02 20:14:24.261883 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.261835 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 20:14:24.261883 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.261847 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d646d5849-btrvm: secret "image-registry-tls" not found Nov 02 20:14:24.261883 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.261883 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls podName:9cb81e57-c7dc-4447-959f-d5496ffee976 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:28.261872149 +0000 UTC m=+38.674007840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls") pod "image-registry-5d646d5849-btrvm" (UID: "9cb81e57-c7dc-4447-959f-d5496ffee976") : secret "image-registry-tls" not found Nov 02 20:14:24.262158 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.261941 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:24.262158 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.261974 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls podName:6fea26b0-2cef-4683-a20f-620ef0880bfc nodeName:}" failed. No retries permitted until 2026-11-02 20:14:28.261962471 +0000 UTC m=+38.674098155 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ndzpm" (UID: "6fea26b0-2cef-4683-a20f-620ef0880bfc") : secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:24.262158 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.262047 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle podName:cb6295a6-f31e-48a3-b4df-7d872a961161 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:28.2620375 +0000 UTC m=+38.674173184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle") pod "router-default-f6f5bd44c-4tksg" (UID: "cb6295a6-f31e-48a3-b4df-7d872a961161") : configmap references non-existent config key: service-ca.crt Nov 02 20:14:24.262158 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.262126 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 20:14:24.262359 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.262222 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert podName:9d5e73f9-907c-40b1-a596-b079ded3abfb nodeName:}" failed. No retries permitted until 2026-11-02 20:14:28.262150839 +0000 UTC m=+38.674286527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-89j9l" (UID: "9d5e73f9-907c-40b1-a596-b079ded3abfb") : secret "networking-console-plugin-cert" not found Nov 02 20:14:24.262359 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.262347 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 20:14:24.262359 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.262359 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6578786844-fgcrv: secret "image-registry-tls" not found Nov 02 20:14:24.262500 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.262392 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls podName:02eded99-edf7-41a0-8866-549252be5fe0 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:28.262381827 +0000 UTC m=+38.674517513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls") pod "image-registry-6578786844-fgcrv" (UID: "02eded99-edf7-41a0-8866-549252be5fe0") : secret "image-registry-tls" not found Nov 02 20:14:24.262500 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.262453 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 20:14:24.262500 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.262484 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs podName:cb6295a6-f31e-48a3-b4df-7d872a961161 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:28.262475099 +0000 UTC m=+38.674610782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs") pod "router-default-f6f5bd44c-4tksg" (UID: "cb6295a6-f31e-48a3-b4df-7d872a961161") : secret "router-metrics-certs-default" not found Nov 02 20:14:24.262625 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.262544 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 20:14:24.262625 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:24.262575 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls podName:8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed nodeName:}" failed. No retries permitted until 2026-11-02 20:14:28.262567179 +0000 UTC m=+38.674702867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-77d6r" (UID: "8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed") : secret "samples-operator-tls" not found Nov 02 20:14:24.284355 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:24.284251 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jjtbq" event={"ID":"afe07c4f-7282-4239-9761-c61ef4153cd9","Type":"ContainerStarted","Data":"fbcf8209bfc27d3f3e1b23824d5dd3abfe29021d719ae75cb9fa223d7f3297dd"} Nov 02 20:14:24.289044 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:24.288211 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-bgd5x" event={"ID":"f22a4aac-bd75-41d2-b1c2-9293bb405389","Type":"ContainerStarted","Data":"a58b4385f1e3b39d20564593eca35c33a96de65ca01c5053701e9378aba1fc4e"} Nov 02 20:14:24.290914 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:24.290829 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rphxh" event={"ID":"f8dd1189-9c04-4ba1-a349-04e7dabcab52","Type":"ContainerStarted","Data":"b6cc565920073e6b826eb68c89265880f87092b390ddbfaf0f49c582b108cb60"} Nov 02 20:14:24.399366 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:24.399311 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-bgd5x" podStartSLOduration=352.632283066 podStartE2EDuration="5m55.399293237s" totalImagesPullingTime="2.767010171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:24.398046891 +0000 UTC m=+34.810182658" watchObservedRunningTime="2026-11-02 20:14:24.399293237 +0000 UTC m=+34.811428943" Nov 02 20:14:24.634110 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:24.634057 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:24.917306 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:24.917221 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:26.303110 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:26.302373 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/0.log" Nov 02 20:14:26.303110 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:26.302413 2576 generic.go:356] "Generic (PLEG): container finished" podID="a6fea55b-74c2-40b1-9952-cd767e448317" containerID="d02a69610d781246e1f02b3fdd5b54cf5e3fcd97b358ff4c76c7301adeaad803" exitCode=255 Nov 02 20:14:26.303110 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:26.302475 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" event={"ID":"a6fea55b-74c2-40b1-9952-cd767e448317","Type":"ContainerDied","Data":"d02a69610d781246e1f02b3fdd5b54cf5e3fcd97b358ff4c76c7301adeaad803"} Nov 02 20:14:26.303110 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:26.302828 2576 scope.go:122] "RemoveContainer" containerID="d02a69610d781246e1f02b3fdd5b54cf5e3fcd97b358ff4c76c7301adeaad803" Nov 02 20:14:27.311539 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:27.311471 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/1.log" Nov 02 20:14:27.312044 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:27.312013 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/0.log" Nov 02 20:14:27.312173 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:27.312058 2576 generic.go:356] "Generic (PLEG): container finished" podID="a6fea55b-74c2-40b1-9952-cd767e448317" containerID="05b60e41e161918b16b6c84ef6c992e2704d297642ef014aee8dfd11bbdfb89b" exitCode=255 Nov 02 20:14:27.312249 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:27.312167 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" event={"ID":"a6fea55b-74c2-40b1-9952-cd767e448317","Type":"ContainerDied","Data":"05b60e41e161918b16b6c84ef6c992e2704d297642ef014aee8dfd11bbdfb89b"} Nov 02 20:14:27.312303 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:27.312266 2576 scope.go:122] "RemoveContainer" containerID="d02a69610d781246e1f02b3fdd5b54cf5e3fcd97b358ff4c76c7301adeaad803" Nov 02 20:14:27.312576 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:27.312556 2576 scope.go:122] "RemoveContainer" containerID="05b60e41e161918b16b6c84ef6c992e2704d297642ef014aee8dfd11bbdfb89b" Nov 02 20:14:27.312791 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:27.312764 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2kxv2_openshift-console-operator(a6fea55b-74c2-40b1-9952-cd767e448317)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" podUID="a6fea55b-74c2-40b1-9952-cd767e448317" Nov 02 20:14:27.314343 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:27.314322 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-v9wrk" event={"ID":"f65dc226-70a9-4fcf-ae3b-28559c2ff235","Type":"ContainerStarted","Data":"154c208bc990652c55848959ffa45522d8d88c087f9e091c5dec3a7bca8587f8"} Nov 02 20:14:27.539861 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:27.539802 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-v9wrk" podStartSLOduration=350.824749603 podStartE2EDuration="5m56.539753657s" totalImagesPullingTime="5.715004054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:27.538184315 +0000 UTC m=+37.950320026" watchObservedRunningTime="2026-11-02 20:14:27.539753657 +0000 UTC m=+37.951889360" Nov 02 20:14:28.311050 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.311010 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:28.311260 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.311077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:28.311260 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.311125 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:28.311260 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311200 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 20:14:28.311260 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311221 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d646d5849-btrvm: secret "image-registry-tls" not found Nov 02 20:14:28.311260 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.311229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-89j9l\" (UID: \"9d5e73f9-907c-40b1-a596-b079ded3abfb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:28.311260 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311243 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle podName:cb6295a6-f31e-48a3-b4df-7d872a961161 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:36.311225753 +0000 UTC m=+46.723361463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle") pod "router-default-f6f5bd44c-4tksg" (UID: "cb6295a6-f31e-48a3-b4df-7d872a961161") : configmap references non-existent config key: service-ca.crt Nov 02 20:14:28.311587 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311272 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls podName:9cb81e57-c7dc-4447-959f-d5496ffee976 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:36.311256336 +0000 UTC m=+46.723392039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls") pod "image-registry-5d646d5849-btrvm" (UID: "9cb81e57-c7dc-4447-959f-d5496ffee976") : secret "image-registry-tls" not found Nov 02 20:14:28.311587 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311289 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 20:14:28.311587 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311197 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:28.311587 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311323 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert podName:9d5e73f9-907c-40b1-a596-b079ded3abfb nodeName:}" failed. No retries permitted until 2026-11-02 20:14:36.311312181 +0000 UTC m=+46.723447865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-89j9l" (UID: "9d5e73f9-907c-40b1-a596-b079ded3abfb") : secret "networking-console-plugin-cert" not found Nov 02 20:14:28.311587 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.311361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:28.311587 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.311390 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:28.311587 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.311429 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:28.311587 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.311455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:28.311587 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311474 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 20:14:28.311587 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311485 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6578786844-fgcrv: secret "image-registry-tls" not found Nov 02 20:14:28.311587 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311516 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls podName:02eded99-edf7-41a0-8866-549252be5fe0 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:36.311504428 +0000 UTC m=+46.723640114 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls") pod "image-registry-6578786844-fgcrv" (UID: "02eded99-edf7-41a0-8866-549252be5fe0") : secret "image-registry-tls" not found Nov 02 20:14:28.311587 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311551 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 20:14:28.311587 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311565 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 20:14:28.311587 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311587 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls podName:6fea26b0-2cef-4683-a20f-620ef0880bfc nodeName:}" failed. No retries permitted until 2026-11-02 20:14:36.311571481 +0000 UTC m=+46.723707170 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ndzpm" (UID: "6fea26b0-2cef-4683-a20f-620ef0880bfc") : secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:28.312578 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311608 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls podName:283c8ed3-38ac-4a45-82d0-ec5bd9245376 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:36.311597201 +0000 UTC m=+46.723732895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls") pod "dns-default-8lr5b" (UID: "283c8ed3-38ac-4a45-82d0-ec5bd9245376") : secret "dns-default-metrics-tls" not found Nov 02 20:14:28.312578 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311634 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs podName:cb6295a6-f31e-48a3-b4df-7d872a961161 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:36.311626037 +0000 UTC m=+46.723761725 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs") pod "router-default-f6f5bd44c-4tksg" (UID: "cb6295a6-f31e-48a3-b4df-7d872a961161") : secret "router-metrics-certs-default" not found Nov 02 20:14:28.312578 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311635 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 20:14:28.312578 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.311676 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls podName:8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed nodeName:}" failed. No retries permitted until 2026-11-02 20:14:36.311663973 +0000 UTC m=+46.723799661 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-77d6r" (UID: "8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed") : secret "samples-operator-tls" not found Nov 02 20:14:28.320534 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.320514 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/1.log" Nov 02 20:14:28.321027 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.321008 2576 scope.go:122] "RemoveContainer" containerID="05b60e41e161918b16b6c84ef6c992e2704d297642ef014aee8dfd11bbdfb89b" Nov 02 20:14:28.321271 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.321242 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2kxv2_openshift-console-operator(a6fea55b-74c2-40b1-9952-cd767e448317)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" podUID="a6fea55b-74c2-40b1-9952-cd767e448317" Nov 02 20:14:28.323098 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.323049 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-278nr" event={"ID":"99e51c7f-420e-4709-b098-2ac9f0eaaedb","Type":"ContainerStarted","Data":"d7107b3f8ffabfd7007a3679cc6c8f8f291117ec7696ca06861bcfd78570dbaa"} Nov 02 20:14:28.325310 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.325287 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" event={"ID":"896fb858-68d1-462c-8a00-2b2b6b5a01af","Type":"ContainerStarted","Data":"076f669bd772da8292e2d7f35ae135673036b5faa09532ac3b60b45fcb2ba3b7"} Nov 02 20:14:28.509545 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.509505 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l4q8k"] Nov 02 20:14:28.537155 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.536575 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:28.544000 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.542730 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" podStartSLOduration=353.103853595 podStartE2EDuration="5m59.542714204s" totalImagesPullingTime="6.438860609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:28.542018415 +0000 UTC m=+38.954154121" watchObservedRunningTime="2026-11-02 20:14:28.542714204 +0000 UTC m=+38.954849910" Nov 02 20:14:28.585132 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.585050 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l4q8k"] Nov 02 20:14:28.586806 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.586604 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 20:14:28.586806 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.586620 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 20:14:28.586806 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.586738 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d6nbh\"" Nov 02 20:14:28.613555 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.613518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7krck\" (UniqueName: \"kubernetes.io/projected/2c868547-5957-44d5-99d3-4aefc664abb1-kube-api-access-7krck\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:28.613704 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.613618 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c868547-5957-44d5-99d3-4aefc664abb1-crio-socket\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:28.613704 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.613686 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c868547-5957-44d5-99d3-4aefc664abb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:28.613826 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.613728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c868547-5957-44d5-99d3-4aefc664abb1-data-volume\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:28.613826 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.613753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:28.651438 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.651368 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-278nr" podStartSLOduration=2.253080763 podStartE2EDuration="8.651354906s" totalImagesPullingTime="6.398274143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:28.651025902 +0000 UTC m=+39.063161607" watchObservedRunningTime="2026-11-02 20:14:28.651354906 +0000 UTC m=+39.063490610" Nov 02 20:14:28.714384 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.714310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c868547-5957-44d5-99d3-4aefc664abb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:28.714832 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.714810 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 20:14:28.714987 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.714971 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c868547-5957-44d5-99d3-4aefc664abb1-kube-rbac-proxy-cm podName:2c868547-5957-44d5-99d3-4aefc664abb1 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:29.214948147 +0000 UTC m=+39.627083835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c868547-5957-44d5-99d3-4aefc664abb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l4q8k" (UID: "2c868547-5957-44d5-99d3-4aefc664abb1") : configmap "kube-rbac-proxy" not found Nov 02 20:14:28.715072 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.715038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c868547-5957-44d5-99d3-4aefc664abb1-data-volume\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:28.715072 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.715064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:28.715223 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.715208 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 20:14:28.715292 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:28.715279 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls podName:2c868547-5957-44d5-99d3-4aefc664abb1 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:29.215243588 +0000 UTC m=+39.627379288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l4q8k" (UID: "2c868547-5957-44d5-99d3-4aefc664abb1") : secret "insights-runtime-extractor-tls" not found Nov 02 20:14:28.715470 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.715422 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c868547-5957-44d5-99d3-4aefc664abb1-data-volume\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:28.715470 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.715464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7krck\" (UniqueName: \"kubernetes.io/projected/2c868547-5957-44d5-99d3-4aefc664abb1-kube-api-access-7krck\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:28.715653 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.715545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c868547-5957-44d5-99d3-4aefc664abb1-crio-socket\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:28.715794 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.715773 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c868547-5957-44d5-99d3-4aefc664abb1-crio-socket\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:28.777489 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.777460 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7krck\" (UniqueName: \"kubernetes.io/projected/2c868547-5957-44d5-99d3-4aefc664abb1-kube-api-access-7krck\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:28.808501 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.808467 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-hxckx-j8tzv"] Nov 02 20:14:28.830537 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.830509 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:28.865246 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.865214 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hxckx-j8tzv"] Nov 02 20:14:28.880381 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.880345 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9bjnj\"" Nov 02 20:14:28.884656 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.884629 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:28.917406 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.917372 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-trusted-ca-bundle\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:28.917550 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.917462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4d75d87e-395e-4668-8b7a-42bdd79d08d9-archives-path\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:28.917550 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.917523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltqcp\" (UniqueName: \"kubernetes.io/projected/4d75d87e-395e-4668-8b7a-42bdd79d08d9-kube-api-access-ltqcp\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:28.917641 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:28.917600 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-service-ca-bundle\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:29.018992 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:29.018955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-trusted-ca-bundle\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:29.019153 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:29.019022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4d75d87e-395e-4668-8b7a-42bdd79d08d9-archives-path\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:29.019153 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:29.019064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltqcp\" (UniqueName: \"kubernetes.io/projected/4d75d87e-395e-4668-8b7a-42bdd79d08d9-kube-api-access-ltqcp\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:29.019245 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:29.019165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-service-ca-bundle\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:29.019742 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:29.019718 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-service-ca-bundle\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:29.020027 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:29.020007 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4d75d87e-395e-4668-8b7a-42bdd79d08d9-archives-path\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:29.020064 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:29.020007 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-trusted-ca-bundle\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:29.083005 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:29.082969 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltqcp\" (UniqueName: \"kubernetes.io/projected/4d75d87e-395e-4668-8b7a-42bdd79d08d9-kube-api-access-ltqcp\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:29.128852 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:29.128696 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:29.165110 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:29.165063 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:29.222169 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:29.221465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c868547-5957-44d5-99d3-4aefc664abb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:29.222169 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:29.221510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:29.222169 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:29.221714 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 20:14:29.222169 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:29.221763 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls podName:2c868547-5957-44d5-99d3-4aefc664abb1 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:30.22175095 +0000 UTC m=+40.633886647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l4q8k" (UID: "2c868547-5957-44d5-99d3-4aefc664abb1") : secret "insights-runtime-extractor-tls" not found Nov 02 20:14:29.222169 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:29.222067 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 20:14:29.222169 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:29.222142 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c868547-5957-44d5-99d3-4aefc664abb1-kube-rbac-proxy-cm podName:2c868547-5957-44d5-99d3-4aefc664abb1 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:30.222125726 +0000 UTC m=+40.634261418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c868547-5957-44d5-99d3-4aefc664abb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l4q8k" (UID: "2c868547-5957-44d5-99d3-4aefc664abb1") : configmap "kube-rbac-proxy" not found Nov 02 20:14:29.377198 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:29.377163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:29.551726 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:29.551705 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hxckx-j8tzv"] Nov 02 20:14:29.554319 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:14:29.554292 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d75d87e_395e_4668_8b7a_42bdd79d08d9.slice/crio-d6151eb6e612794d3cdae41786b68fae9972d326cb1b3e497e02d5d72f9bf66f WatchSource:0}: Error finding container d6151eb6e612794d3cdae41786b68fae9972d326cb1b3e497e02d5d72f9bf66f: Status 404 returned error can't find the container with id d6151eb6e612794d3cdae41786b68fae9972d326cb1b3e497e02d5d72f9bf66f Nov 02 20:14:29.627677 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:29.627635 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:30.232417 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.232343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c868547-5957-44d5-99d3-4aefc664abb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:30.232417 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.232405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:30.232632 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:30.232466 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 20:14:30.232632 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:30.232511 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 20:14:30.232632 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:30.232545 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c868547-5957-44d5-99d3-4aefc664abb1-kube-rbac-proxy-cm podName:2c868547-5957-44d5-99d3-4aefc664abb1 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:32.232526035 +0000 UTC m=+42.644661735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c868547-5957-44d5-99d3-4aefc664abb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l4q8k" (UID: "2c868547-5957-44d5-99d3-4aefc664abb1") : configmap "kube-rbac-proxy" not found Nov 02 20:14:30.232632 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:30.232570 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls podName:2c868547-5957-44d5-99d3-4aefc664abb1 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:32.232554189 +0000 UTC m=+42.644689887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l4q8k" (UID: "2c868547-5957-44d5-99d3-4aefc664abb1") : secret "insights-runtime-extractor-tls" not found Nov 02 20:14:30.334028 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.333935 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rphxh" event={"ID":"f8dd1189-9c04-4ba1-a349-04e7dabcab52","Type":"ContainerStarted","Data":"509c5dacd90d526f41f85cb42434f56956e3893cf185f09caa3a5fba42228cae"} Nov 02 20:14:30.334028 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.333994 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:30.336251 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.336153 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hxckx-j8tzv" event={"ID":"4d75d87e-395e-4668-8b7a-42bdd79d08d9","Type":"ContainerStarted","Data":"26850d4a4881a050a1e914f96360fcec2e0aaeab155729e24d2844f83e6d91fe"} Nov 02 20:14:30.336251 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.336208 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hxckx-j8tzv" event={"ID":"4d75d87e-395e-4668-8b7a-42bdd79d08d9","Type":"ContainerStarted","Data":"d6151eb6e612794d3cdae41786b68fae9972d326cb1b3e497e02d5d72f9bf66f"} Nov 02 20:14:30.337856 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.337829 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tfzr" event={"ID":"bfe84946-82e9-456a-b81f-af887093749b","Type":"ContainerStarted","Data":"db5e60da857ea784ddc81ad310374dcd82d9f5e5d23834bcdf9071d4873b090d"} Nov 02 20:14:30.339443 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.339421 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" event={"ID":"d23cf40e-53e9-406e-92f8-ce628b1baa68","Type":"ContainerStarted","Data":"ef2a96e5640f307ff6a9eda12e05fe8ca8394e911cfa974de5d6f2217306c907"} Nov 02 20:14:30.445331 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.445270 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rphxh" podStartSLOduration=34.025285855 podStartE2EDuration="40.445252091s" totalImagesPullingTime="6.419966236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:30.44364421 +0000 UTC m=+40.855779916" watchObservedRunningTime="2026-11-02 20:14:30.445252091 +0000 UTC m=+40.857387798" Nov 02 20:14:30.538496 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.538424 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4n9mb"] Nov 02 20:14:30.556310 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.556261 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tfzr" podStartSLOduration=350.895814736 podStartE2EDuration="5m59.556242936s" totalImagesPullingTime="8.6604282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:30.555277374 +0000 UTC m=+40.967413080" watchObservedRunningTime="2026-11-02 20:14:30.556242936 +0000 UTC m=+40.968378642" Nov 02 20:14:30.565797 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.564663 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:30.595552 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.595524 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4n9mb"] Nov 02 20:14:30.614818 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.614793 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 20:14:30.614818 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.614810 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 20:14:30.615020 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.614828 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zgrcn\"" Nov 02 20:14:30.615020 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.614902 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 20:14:30.615020 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.614909 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 20:14:30.635554 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.635527 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nczq\" (UniqueName: \"kubernetes.io/projected/eb120384-9c46-4dbd-bb80-bedf4bbe957c-kube-api-access-5nczq\") pod \"service-ca-5fffc97bdf-4n9mb\" (UID: \"eb120384-9c46-4dbd-bb80-bedf4bbe957c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:30.635689 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.635573 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/eb120384-9c46-4dbd-bb80-bedf4bbe957c-signing-cabundle\") pod \"service-ca-5fffc97bdf-4n9mb\" (UID: \"eb120384-9c46-4dbd-bb80-bedf4bbe957c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:30.635689 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.635594 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/eb120384-9c46-4dbd-bb80-bedf4bbe957c-signing-key\") pod \"service-ca-5fffc97bdf-4n9mb\" (UID: \"eb120384-9c46-4dbd-bb80-bedf4bbe957c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:30.666596 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.666548 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-hxckx-j8tzv" podStartSLOduration=2.6665349210000002 podStartE2EDuration="2.666534921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:30.665221579 +0000 UTC m=+41.077357287" watchObservedRunningTime="2026-11-02 20:14:30.666534921 +0000 UTC m=+41.078670627" Nov 02 20:14:30.737021 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.736995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/eb120384-9c46-4dbd-bb80-bedf4bbe957c-signing-cabundle\") pod \"service-ca-5fffc97bdf-4n9mb\" (UID: \"eb120384-9c46-4dbd-bb80-bedf4bbe957c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:30.737170 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.737036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/eb120384-9c46-4dbd-bb80-bedf4bbe957c-signing-key\") pod \"service-ca-5fffc97bdf-4n9mb\" (UID: \"eb120384-9c46-4dbd-bb80-bedf4bbe957c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:30.737323 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.737305 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nczq\" (UniqueName: \"kubernetes.io/projected/eb120384-9c46-4dbd-bb80-bedf4bbe957c-kube-api-access-5nczq\") pod \"service-ca-5fffc97bdf-4n9mb\" (UID: \"eb120384-9c46-4dbd-bb80-bedf4bbe957c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:30.737747 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.737723 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/eb120384-9c46-4dbd-bb80-bedf4bbe957c-signing-cabundle\") pod \"service-ca-5fffc97bdf-4n9mb\" (UID: \"eb120384-9c46-4dbd-bb80-bedf4bbe957c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:30.741113 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.740048 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/eb120384-9c46-4dbd-bb80-bedf4bbe957c-signing-key\") pod \"service-ca-5fffc97bdf-4n9mb\" (UID: \"eb120384-9c46-4dbd-bb80-bedf4bbe957c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:30.781180 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.781127 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" podStartSLOduration=351.585528192 podStartE2EDuration="5m59.781110529s" totalImagesPullingTime="8.195582337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:30.779034424 +0000 UTC m=+41.191170162" watchObservedRunningTime="2026-11-02 20:14:30.781110529 +0000 UTC m=+41.193246233" Nov 02 20:14:30.797436 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.797379 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nczq\" (UniqueName: \"kubernetes.io/projected/eb120384-9c46-4dbd-bb80-bedf4bbe957c-kube-api-access-5nczq\") pod \"service-ca-5fffc97bdf-4n9mb\" (UID: \"eb120384-9c46-4dbd-bb80-bedf4bbe957c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:30.805620 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.805598 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:30.805753 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.805662 2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:30.806110 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.806065 2576 scope.go:122] "RemoveContainer" containerID="05b60e41e161918b16b6c84ef6c992e2704d297642ef014aee8dfd11bbdfb89b" Nov 02 20:14:30.806376 ip-10-0-140-120 kubenswrapper[2576]: E1102 20:14:30.806355 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2kxv2_openshift-console-operator(a6fea55b-74c2-40b1-9952-cd767e448317)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" podUID="a6fea55b-74c2-40b1-9952-cd767e448317" Nov 02 20:14:30.870324 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.870290 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:30.892605 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:30.892567 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:31.113155 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:31.113121 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:31.288111 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:31.288065 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4n9mb"] Nov 02 20:14:31.289989 ip-10-0-140-120 kubenswrapper[2576]: W1102 20:14:31.289954 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb120384_9c46_4dbd_bb80_bedf4bbe957c.slice/crio-636ed99fad21ca95ddac9d9ea7049bbaec0c1cae39d728cbf0aff5d11b0dd426 WatchSource:0}: Error finding container 636ed99fad21ca95ddac9d9ea7049bbaec0c1cae39d728cbf0aff5d11b0dd426: Status 404 returned error can't find the container with id 636ed99fad21ca95ddac9d9ea7049bbaec0c1cae39d728cbf0aff5d11b0dd426 Nov 02 20:14:31.344865 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:31.344839 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" event={"ID":"eb120384-9c46-4dbd-bb80-bedf4bbe957c","Type":"ContainerStarted","Data":"636ed99fad21ca95ddac9d9ea7049bbaec0c1cae39d728cbf0aff5d11b0dd426"} Nov 02 20:14:31.356456 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:31.356427 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:31.628050 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:31.628018 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:31.649180 ip-10-0-140-120 kubenswrapper[2576]: I1102 20:14:31.649039 2576 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 20:14:31.649275 ip-10-0-140-120 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 20:14:31.664680 ip-10-0-140-120 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 20:14:31.664888 ip-10-0-140-120 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 20:14:31.664942 ip-10-0-140-120 systemd[1]: kubelet.service: Consumed 3.028s CPU time. Nov 02 20:14:31.678775 ip-10-0-140-120 systemd[1]: Starting Kubernetes Kubelet... Nov 02 20:14:31.774968 ip-10-0-140-120 kubenswrapper[4971]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 20:14:31.774968 ip-10-0-140-120 kubenswrapper[4971]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 20:14:31.774968 ip-10-0-140-120 kubenswrapper[4971]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 20:14:31.774968 ip-10-0-140-120 kubenswrapper[4971]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 20:14:31.776960 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.776879 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:14:31.776960 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.776953 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:14:31.776960 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.776958 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:14:31.776960 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.776960 4971 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:14:31.776960 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.776963 4971 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:14:31.776960 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.776966 4971 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:14:31.776960 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.776970 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.776973 4971 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.776975 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.776978 4971 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.776981 4971 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.776984 4971 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.776987 4971 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.776989 4971 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777001 4971 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777004 4971 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777007 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777009 4971 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777011 4971 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777014 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777016 4971 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777019 4971 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777022 4971 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777024 4971 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777027 4971 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777029 4971 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:14:31.777321 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777032 4971 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777034 4971 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777037 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777040 4971 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777042 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777045 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777048 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777052 4971 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777058 4971 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777065 4971 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777070 4971 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777074 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777104 4971 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777109 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777114 4971 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777119 4971 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777124 4971 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777128 4971 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777131 4971 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:14:31.777795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777134 4971 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777137 4971 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777141 4971 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777143 4971 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777146 4971 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777149 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777152 4971 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777154 4971 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777157 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777162 4971 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777166 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777169 4971 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777171 4971 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777174 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777176 4971 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777179 4971 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777182 4971 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777184 4971 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:14:31.778286 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777187 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777189 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777192 4971 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777194 4971 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777197 4971 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777200 4971 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777202 4971 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777205 4971 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777207 4971 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777210 4971 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777213 4971 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777215 4971 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777217 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777228 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777231 4971 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777234 4971 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777236 4971 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777239 4971 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777242 4971 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777245 4971 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:14:31.778721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777247 4971 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777250 4971 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777253 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777255 4971 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777258 4971 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777260 4971 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777263 4971 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777265 4971 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777268 4971 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777270 4971 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777273 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777277 4971 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777279 4971 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777282 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777285 4971 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777287 4971 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777289 4971 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777292 4971 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777295 4971 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777297 4971 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:14:31.779274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777299 4971 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777302 4971 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.777304 4971 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778033 4971 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778040 4971 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778042 4971 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778045 4971 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778048 4971 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778051 4971 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778054 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778057 4971 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778059 4971 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778062 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778065 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778067 4971 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778069 4971 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778072 4971 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778075 4971 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778100 4971 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:14:31.779770 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778109 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778113 4971 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778116 4971 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778128 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778132 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778135 4971 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778137 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778141 4971 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778144 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778148 4971 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778152 4971 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778154 4971 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778157 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778159 4971 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778162 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778164 4971 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778167 4971 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778169 4971 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778171 4971 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778174 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:14:31.780236 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778176 4971 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778180 4971 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778183 4971 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778186 4971 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778188 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778191 4971 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778195 4971 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778198 4971 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778200 4971 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778203 4971 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778205 4971 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778208 4971 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778210 4971 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778213 4971 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778215 4971 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778218 4971 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778220 4971 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778223 4971 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778225 4971 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778227 4971 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:14:31.780725 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778230 4971 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778233 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778235 4971 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778237 4971 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778240 4971 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778242 4971 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778245 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778248 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778250 4971 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778252 4971 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778255 4971 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778257 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778261 4971 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778264 4971 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778266 4971 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778269 4971 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778272 4971 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778274 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778278 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:14:31.781232 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778281 4971 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778283 4971 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778286 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778289 4971 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778291 4971 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778294 4971 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778296 4971 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778299 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778302 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778304 4971 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778307 4971 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778309 4971 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778312 4971 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778315 4971 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778317 4971 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778319 4971 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778322 4971 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778325 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778327 4971 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778329 4971 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:14:31.781718 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778333 4971 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778335 4971 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778338 4971 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778340 4971 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778343 4971 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778345 4971 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778348 4971 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778350 4971 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778353 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778355 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778358 4971 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778854 4971 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778860 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778863 4971 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778867 4971 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778870 4971 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778873 4971 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778875 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778878 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778881 4971 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:14:31.782210 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778884 4971 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778886 4971 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778889 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778891 4971 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778894 4971 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778897 4971 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778899 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778902 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778904 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778907 4971 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778909 4971 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778912 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778915 4971 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778917 4971 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778920 4971 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778923 4971 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778925 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778927 4971 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778930 4971 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:14:31.782701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778932 4971 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778935 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778938 4971 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778940 4971 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778944 4971 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778947 4971 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778949 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778952 4971 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778954 4971 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778956 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778959 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778962 4971 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778964 4971 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778966 4971 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778969 4971 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778971 4971 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778974 4971 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778976 4971 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778978 4971 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778981 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:14:31.783185 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778983 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778987 4971 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778989 4971 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778993 4971 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778997 4971 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.778999 4971 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779002 4971 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779004 4971 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779007 4971 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779009 4971 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779012 4971 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779016 4971 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779020 4971 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779023 4971 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779026 4971 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779029 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779032 4971 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779035 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779037 4971 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:14:31.783678 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779040 4971 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779042 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779044 4971 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779047 4971 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779049 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779052 4971 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779054 4971 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779057 4971 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779060 4971 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779062 4971 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779064 4971 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779067 4971 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779070 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779072 4971 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779075 4971 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779096 4971 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779099 4971 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779102 4971 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779104 4971 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779107 4971 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:14:31.784144 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779109 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779112 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779115 4971 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779117 4971 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779121 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779123 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779126 4971 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779128 4971 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779131 4971 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779134 4971 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779136 4971 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779139 4971 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779141 4971 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779144 4971 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779146 4971 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779149 4971 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779152 4971 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779154 4971 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779157 4971 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779259 4971 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 20:14:31.784619 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779278 4971 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779287 4971 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779293 4971 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779300 4971 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779304 4971 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779309 4971 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779317 4971 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779320 4971 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779323 4971 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779327 4971 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779330 4971 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779333 4971 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779336 4971 flags.go:64] FLAG: --cgroup-root="" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779339 4971 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779341 4971 flags.go:64] FLAG: --client-ca-file="" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779344 4971 flags.go:64] FLAG: --cloud-provider="external" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779347 4971 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779352 4971 flags.go:64] FLAG: --cluster-domain="" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779354 4971 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779357 4971 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779361 4971 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779364 4971 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779369 4971 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 20:14:31.785146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779371 4971 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779374 4971 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779377 4971 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779380 4971 flags.go:64] FLAG: --contention-profiling="false" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779383 4971 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779386 4971 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779389 4971 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779392 4971 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779396 4971 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779399 4971 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779401 4971 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779404 4971 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779407 4971 flags.go:64] FLAG: --enable-server="true" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779410 4971 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779414 4971 flags.go:64] FLAG: --event-burst="100" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779416 4971 flags.go:64] FLAG: --event-qps="50" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779419 4971 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779422 4971 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779425 4971 flags.go:64] FLAG: --eviction-hard="" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779429 4971 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779432 4971 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779435 4971 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779437 4971 flags.go:64] FLAG: --eviction-soft="" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779440 4971 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779443 4971 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 20:14:31.785693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779450 4971 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779453 4971 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779456 4971 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779458 4971 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779461 4971 flags.go:64] FLAG: --feature-gates="" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779465 4971 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779468 4971 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779472 4971 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779475 4971 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779478 4971 flags.go:64] FLAG: --healthz-port="10248" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779481 4971 flags.go:64] FLAG: --help="false" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779484 4971 flags.go:64] FLAG: --hostname-override="ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779487 4971 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779490 4971 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779493 4971 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779496 4971 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779500 4971 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779503 4971 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779505 4971 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779508 4971 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779511 4971 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779514 4971 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779517 4971 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779519 4971 flags.go:64] FLAG: --kube-reserved="" Nov 02 20:14:31.786534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779522 4971 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779525 4971 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779528 4971 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779531 4971 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779533 4971 flags.go:64] FLAG: --lock-file="" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779536 4971 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779539 4971 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779542 4971 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779550 4971 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779553 4971 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779556 4971 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779559 4971 flags.go:64] FLAG: --logging-format="text" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779562 4971 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779565 4971 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779568 4971 flags.go:64] FLAG: --manifest-url="" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779571 4971 flags.go:64] FLAG: --manifest-url-header="" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779575 4971 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779578 4971 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779582 4971 flags.go:64] FLAG: --max-pods="110" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779585 4971 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779588 4971 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779591 4971 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779593 4971 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779596 4971 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779599 4971 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 20:14:31.787118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779602 4971 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779609 4971 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779612 4971 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779615 4971 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779618 4971 flags.go:64] FLAG: --pod-cidr="" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779621 4971 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779623 4971 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779626 4971 flags.go:64] FLAG: --pods-per-core="0" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779629 4971 flags.go:64] FLAG: --port="10250" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779632 4971 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779635 4971 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0ab8ed626e5ec6847" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779638 4971 flags.go:64] FLAG: --qos-reserved="" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779640 4971 flags.go:64] FLAG: --read-only-port="10255" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779643 4971 flags.go:64] FLAG: --register-node="true" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779646 4971 flags.go:64] FLAG: --register-with-taints="" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779652 4971 flags.go:64] FLAG: --registry-burst="10" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779655 4971 flags.go:64] FLAG: --registry-qps="5" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779658 4971 flags.go:64] FLAG: --reserved-cpus="" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779661 4971 flags.go:64] FLAG: --reserved-memory="" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779664 4971 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779667 4971 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779670 4971 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779673 4971 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779676 4971 flags.go:64] FLAG: --runonce="false" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779679 4971 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779682 4971 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779685 4971 flags.go:64] FLAG: --seccomp-default="false" Nov 02 20:14:31.787751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779688 4971 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779690 4971 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779693 4971 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779696 4971 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779699 4971 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779702 4971 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779704 4971 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779707 4971 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779710 4971 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779713 4971 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779715 4971 flags.go:64] FLAG: --system-cgroups="" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779718 4971 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779723 4971 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779726 4971 flags.go:64] FLAG: --tls-cert-file="" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779729 4971 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779733 4971 flags.go:64] FLAG: --tls-min-version="" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779735 4971 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779738 4971 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779741 4971 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779743 4971 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779750 4971 flags.go:64] FLAG: --v="2" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779755 4971 flags.go:64] FLAG: --version="false" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779759 4971 flags.go:64] FLAG: --vmodule="" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779762 4971 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 20:14:31.788444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.779765 4971 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779890 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779895 4971 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779898 4971 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779901 4971 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779903 4971 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779906 4971 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779909 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779912 4971 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779914 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779917 4971 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779919 4971 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779922 4971 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779925 4971 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779927 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779929 4971 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779932 4971 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779934 4971 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779937 4971 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:14:31.789357 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779939 4971 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779941 4971 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779944 4971 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779946 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779950 4971 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779953 4971 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779956 4971 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779959 4971 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779961 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779967 4971 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779970 4971 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779972 4971 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779974 4971 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779977 4971 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779979 4971 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779982 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779984 4971 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779987 4971 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779989 4971 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:14:31.790143 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779991 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779994 4971 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.779997 4971 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780000 4971 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780002 4971 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780005 4971 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780007 4971 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780009 4971 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780012 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780014 4971 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780017 4971 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780019 4971 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780022 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780024 4971 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780028 4971 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780031 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780034 4971 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780037 4971 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780040 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:14:31.790745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780042 4971 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780045 4971 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780047 4971 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780052 4971 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780055 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780057 4971 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780060 4971 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780062 4971 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780065 4971 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780067 4971 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780070 4971 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780072 4971 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780075 4971 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780093 4971 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780095 4971 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780098 4971 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780101 4971 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780103 4971 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780106 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780108 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:14:31.791414 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780111 4971 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780114 4971 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780116 4971 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780119 4971 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780121 4971 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780124 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780126 4971 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780129 4971 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780131 4971 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780133 4971 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780136 4971 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780138 4971 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780141 4971 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780143 4971 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780146 4971 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780151 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780153 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780156 4971 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780158 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780160 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:14:31.792319 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780163 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:14:31.792879 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780165 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:14:31.792879 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780168 4971 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:14:31.792879 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780170 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:14:31.792879 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780173 4971 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:14:31.792879 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780175 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:14:31.792879 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780178 4971 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:14:31.792879 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780181 4971 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:14:31.792879 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780184 4971 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:14:31.792879 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780186 4971 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:14:31.792879 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.780212 4971 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 20:14:31.792879 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780252 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:14:31.792879 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780256 4971 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:14:31.792879 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780259 4971 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:14:31.792879 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780262 4971 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:14:31.792879 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780265 4971 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:14:31.792879 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780267 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780270 4971 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780272 4971 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780275 4971 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780277 4971 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780280 4971 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780282 4971 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780284 4971 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780287 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780291 4971 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780295 4971 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780299 4971 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780303 4971 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780305 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780308 4971 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780310 4971 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780313 4971 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780315 4971 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780318 4971 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:14:31.793539 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780320 4971 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780323 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780325 4971 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780328 4971 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780330 4971 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780333 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780336 4971 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780338 4971 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780341 4971 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780343 4971 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780345 4971 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780348 4971 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780350 4971 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780353 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780355 4971 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780358 4971 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780360 4971 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780363 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780365 4971 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:14:31.794402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780369 4971 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780373 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780376 4971 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780379 4971 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780381 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780385 4971 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780388 4971 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780390 4971 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780393 4971 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780396 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780398 4971 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780401 4971 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780403 4971 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780406 4971 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780408 4971 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780411 4971 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780413 4971 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780416 4971 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780418 4971 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780421 4971 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:14:31.794970 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780423 4971 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780426 4971 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780428 4971 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780431 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780433 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780437 4971 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780439 4971 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780442 4971 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780444 4971 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780447 4971 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780449 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780451 4971 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780454 4971 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780456 4971 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780460 4971 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780462 4971 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780465 4971 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780468 4971 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780472 4971 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780474 4971 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:14:31.795691 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780477 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780479 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780482 4971 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780484 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780487 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780490 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780492 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780495 4971 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780498 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780500 4971 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780503 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780506 4971 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780508 4971 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780511 4971 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780514 4971 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780517 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780519 4971 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780522 4971 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780524 4971 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780527 4971 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:14:31.796219 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780529 4971 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780532 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.780535 4971 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.784878 4971 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.784894 4971 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.784922 4971 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.784928 4971 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785000 4971 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785005 4971 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785008 4971 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785011 4971 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785014 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785017 4971 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785020 4971 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785022 4971 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785025 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785027 4971 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785030 4971 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785032 4971 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785035 4971 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:14:31.796700 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785037 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785040 4971 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785043 4971 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785045 4971 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785047 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785050 4971 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785053 4971 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785055 4971 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785058 4971 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785060 4971 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785062 4971 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785065 4971 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785068 4971 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785070 4971 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785073 4971 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785075 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785099 4971 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785103 4971 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785106 4971 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:14:31.797195 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785108 4971 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785111 4971 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785113 4971 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785116 4971 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785118 4971 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785121 4971 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785125 4971 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785128 4971 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785131 4971 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785134 4971 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785136 4971 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785139 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785141 4971 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785149 4971 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785152 4971 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785155 4971 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785158 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785160 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785163 4971 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785166 4971 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:14:31.797659 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785168 4971 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785171 4971 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785174 4971 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785176 4971 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785179 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785181 4971 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785184 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785186 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785189 4971 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785191 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785200 4971 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785203 4971 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785206 4971 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785208 4971 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785211 4971 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785214 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785216 4971 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785219 4971 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785221 4971 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785223 4971 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:14:31.798237 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785226 4971 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785228 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785231 4971 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785233 4971 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785235 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785238 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785240 4971 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785243 4971 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785245 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785247 4971 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785250 4971 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785252 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785255 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785257 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785259 4971 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785262 4971 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785264 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785267 4971 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785269 4971 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:14:31.798731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785272 4971 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:14:31.799323 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785274 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:14:31.799323 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785278 4971 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:14:31.799323 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785281 4971 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:14:31.799323 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785289 4971 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:14:31.799323 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785293 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:14:31.799323 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785296 4971 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:14:31.799323 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785298 4971 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:14:31.799323 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785301 4971 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:14:31.799323 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785303 4971 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:14:31.799323 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785305 4971 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:14:31.799323 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785308 4971 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:14:31.799323 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785310 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:14:31.799323 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785313 4971 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:14:31.799323 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785315 4971 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:14:31.799323 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.785343 4971 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 20:14:31.799323 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785534 4971 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785538 4971 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785541 4971 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785543 4971 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785546 4971 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785548 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785551 4971 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785553 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785556 4971 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785558 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785561 4971 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785563 4971 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785566 4971 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785568 4971 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785571 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785573 4971 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785575 4971 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785578 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785580 4971 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:14:31.799824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785583 4971 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785585 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785594 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785597 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785599 4971 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785602 4971 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785604 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785606 4971 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785609 4971 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785611 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785614 4971 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785616 4971 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785619 4971 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785621 4971 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785623 4971 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785626 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785628 4971 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785631 4971 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785633 4971 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785635 4971 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:14:31.800397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785638 4971 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785640 4971 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785643 4971 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785645 4971 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785648 4971 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785650 4971 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785652 4971 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785656 4971 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785659 4971 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785662 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785664 4971 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785667 4971 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785669 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785671 4971 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785674 4971 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785681 4971 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785684 4971 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785687 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785689 4971 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:14:31.800986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785691 4971 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785694 4971 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785697 4971 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785699 4971 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785702 4971 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785704 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785707 4971 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785709 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785712 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785714 4971 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785717 4971 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785719 4971 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785721 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785724 4971 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785726 4971 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785729 4971 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785731 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785734 4971 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785736 4971 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:14:31.801697 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785739 4971 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785741 4971 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785744 4971 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785746 4971 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785749 4971 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785751 4971 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785754 4971 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785756 4971 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785758 4971 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785761 4971 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785769 4971 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785771 4971 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785774 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785779 4971 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785782 4971 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785784 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785823 4971 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785827 4971 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785830 4971 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785832 4971 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:14:31.802397 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785835 4971 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:14:31.802989 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785839 4971 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:14:31.802989 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785841 4971 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:14:31.802989 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785844 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:14:31.802989 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785846 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:14:31.802989 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785849 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:14:31.802989 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785851 4971 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:14:31.802989 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785854 4971 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:14:31.802989 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785856 4971 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:14:31.802989 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.785880 4971 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 20:14:31.802989 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785929 4971 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:14:31.802989 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785935 4971 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:14:31.802989 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785937 4971 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:14:31.802989 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785940 4971 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:14:31.802989 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785942 4971 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:14:31.802989 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785945 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:14:31.802989 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785947 4971 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785950 4971 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785952 4971 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785955 4971 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785958 4971 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785960 4971 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785963 4971 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785972 4971 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785975 4971 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785980 4971 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785986 4971 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785992 4971 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785996 4971 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.785999 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786002 4971 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786005 4971 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786007 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786010 4971 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786012 4971 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:14:31.803400 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786015 4971 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786017 4971 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786020 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786022 4971 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786025 4971 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786027 4971 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786030 4971 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786034 4971 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786037 4971 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786039 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786042 4971 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786044 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786047 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786049 4971 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786051 4971 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786055 4971 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786059 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786064 4971 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786068 4971 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786071 4971 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:14:31.803888 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786073 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786097 4971 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786101 4971 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786105 4971 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786107 4971 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786110 4971 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786113 4971 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786115 4971 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786118 4971 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786128 4971 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786132 4971 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786144 4971 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786149 4971 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786153 4971 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786156 4971 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786158 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786161 4971 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786163 4971 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786166 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:14:31.804434 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786171 4971 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786175 4971 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786178 4971 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786181 4971 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786184 4971 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786187 4971 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786190 4971 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786193 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786196 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786198 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786200 4971 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786203 4971 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786205 4971 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786208 4971 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786210 4971 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786223 4971 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786230 4971 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786233 4971 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786235 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:14:31.805002 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786238 4971 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786240 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786243 4971 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786245 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786247 4971 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786250 4971 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786252 4971 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786255 4971 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786257 4971 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786260 4971 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786262 4971 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786265 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786267 4971 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786271 4971 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786273 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786276 4971 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786278 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786281 4971 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786283 4971 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786286 4971 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:14:31.805591 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786288 4971 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:14:31.806254 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786293 4971 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:14:31.806254 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:31.786297 4971 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:14:31.806254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.786360 4971 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 20:14:31.806254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.787350 4971 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 20:14:31.806254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.787412 4971 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 20:14:31.806254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.787617 4971 server.go:1014] "Starting client certificate rotation" Nov 02 20:14:31.806254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.787756 4971 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 20:14:31.806254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.787808 4971 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 20:08:50 +0000 UTC" deadline="2028-03-28 05:34:26.504371663 +0000 UTC" Nov 02 20:14:31.806254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.787843 4971 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12273h19m54.716530688s" Nov 02 20:14:31.806254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.788196 4971 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 20:14:31.806254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.790734 4971 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 20:14:31.806254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.792515 4971 log.go:25] "Validated CRI v1 runtime API" Nov 02 20:14:31.806254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.794496 4971 log.go:25] "Validated CRI v1 image API" Nov 02 20:14:31.806254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.795275 4971 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 20:14:31.806254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.797597 4971 fs.go:135] Filesystem UUIDs: map[01f4dad2-c4e6-4e5e-aa91-bb591973efe6:/dev/nvme0n1p4 19bdc55b-ed72-4c68-8cc1-fef76173ac9b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 02 20:14:31.806780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.797621 4971 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07ab293df918258cd3b264ce3c4eedc9958d2dae6b334a34d3a6e19c5c41af80/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07ab293df918258cd3b264ce3c4eedc9958d2dae6b334a34d3a6e19c5c41af80/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14e005f90d52a2dc6b1bcb210c65223609ecf4699f6910ad0cf71b52d3e38265/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14e005f90d52a2dc6b1bcb210c65223609ecf4699f6910ad0cf71b52d3e38265/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15606ef9514cdd2b5c88060ac28788605d80266d1a0ec4a2b215b17077900d8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15606ef9514cdd2b5c88060ac28788605d80266d1a0ec4a2b215b17077900d8d/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/612faca9a4be33f531f8cf9d8762516888219f40744f562cdb9035696360431c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/612faca9a4be33f531f8cf9d8762516888219f40744f562cdb9035696360431c/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/636ed99fad21ca95ddac9d9ea7049bbaec0c1cae39d728cbf0aff5d11b0dd426/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/636ed99fad21ca95ddac9d9ea7049bbaec0c1cae39d728cbf0aff5d11b0dd426/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/690a5c20ac6d5a4dc1bf8a9ccf0448d9b7d97f3c5d154b65e2b20a6534b579e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/690a5c20ac6d5a4dc1bf8a9ccf0448d9b7d97f3c5d154b65e2b20a6534b579e6/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c7aeeaa257f53ea9aa8d328684a6c694a927c24739b9dbd19481ec1d8ec2d30/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c7aeeaa257f53ea9aa8d328684a6c694a927c24739b9dbd19481ec1d8ec2d30/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71c7ced32f7d40df0fdd5e71f6234eda39d1c2a35f033827b6472f75ccd11856/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71c7ced32f7d40df0fdd5e71f6234eda39d1c2a35f033827b6472f75ccd11856/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7873bc74331e14113791f9d5c98f8803f2300f77f1296ca2975e178b121fb289/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7873bc74331e14113791f9d5c98f8803f2300f77f1296ca2975e178b121fb289/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7bf224f0387a6c1763335ce89ce4f008842b70867aa913d224958400e5e831fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7bf224f0387a6c1763335ce89ce4f008842b70867aa913d224958400e5e831fe/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82c2cb92ed201c94477cd9681fb845f1e683c68a36351893b5b706f136e061c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82c2cb92ed201c94477cd9681fb845f1e683c68a36351893b5b706f136e061c1/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ea136b3030a2d7db3d157c0d72d2877085de68f36077b113c39ef0a8d689e27/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ea136b3030a2d7db3d157c0d72d2877085de68f36077b113c39ef0a8d689e27/userdata/shm major:0 minor:52 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f614f8ecc585a5a2ded495c5f6f82ce305420b1a56eba73a519e203e8c03310/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f614f8ecc585a5a2ded495c5f6f82ce305420b1a56eba73a519e203e8c03310/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa2af273ee991d426f21609d08d24b63d88a5cd6378432b8dc7f8ff622fd7937/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa2af273ee991d426f21609d08d24b63d88a5cd6378432b8dc7f8ff622fd7937/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af81f493cee046ae85b7be06bade3ce4506994eba8b5dda4076bf6413296b13b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af81f493cee046ae85b7be06bade3ce4506994eba8b5dda4076bf6413296b13b/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6cc565920073e6b826eb68c89265880f87092b390ddbfaf0f49c582b108cb60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6cc565920073e6b826eb68c89265880f87092b390ddbfaf0f49c582b108cb60/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc66d97e073c8610d57e2e9b89438cf1dd411d9b8cfa417c158b770ee2cdf1cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc66d97e073c8610d57e2e9b89438cf1dd411d9b8cfa417c158b770ee2cdf1cf/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd0d3d1b1070f9fd7a896f2b22158a6cc02853c2bc49691f7dbd0c1fda88b9cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd0d3d1b1070f9fd7a896f2b22158a6cc02853c2bc49691f7dbd0c1fda88b9cc/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca238d1beee6588afe58ce0417b419e77f5fc5f70fda1e92edaef7c4d552b662/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca238d1beee6588afe58ce0417b419e77f5fc5f70fda1e92edaef7c4d552b662/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbc0f8137d19d2acb1fda8e0914da1d28884aabc4d694a37da80cd622155e926/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbc0f8137d19d2acb1fda8e0914da1d28884aabc4d694a37da80cd622155e926/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6151eb6e612794d3cdae41786b68fae9972d326cb1b3e497e02d5d72f9bf66f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6151eb6e612794d3cdae41786b68fae9972d326cb1b3e497e02d5d72f9bf66f/userdata/shm major:0 minor:322 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee3b20c44102840b9bf245e66c89b2e0941ff66d2bcd0140c2f3d052154e3f71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee3b20c44102840b9bf245e66c89b2e0941ff66d2bcd0140c2f3d052154e3f71/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f34649d86715069ceade5514d1e1fc853e5db5d7b431b9df51b5d232244de298/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f34649d86715069ceade5514d1e1fc853e5db5d7b431b9df51b5d232244de298/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fbcf8209bfc27d3f3e1b23824d5dd3abfe29021d719ae75cb9fa223d7f3297dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fbcf8209bfc27d3f3e1b23824d5dd3abfe29021d719ae75cb9fa223d7f3297dd/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02eded99-edf7-41a0-8866-549252be5fe0/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/02eded99-edf7-41a0-8866-549252be5fe0/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02eded99-edf7-41a0-8866-549252be5fe0/volumes/kubernetes.io~projected/kube-api-access-kmsvc:{mountpoint:/var/lib/kubelet/pods/02eded99-edf7-41a0-8866-549252be5fe0/volumes/kubernetes.io~projected/kube-api-access-kmsvc major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02eded99-edf7-41a0-8866-549252be5fe0/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/02eded99-edf7-41a0-8866-549252be5fe0/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02eded99-edf7-41a0-8866-549252be5fe0/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/02eded99-edf7-41a0-8866-549252be5fe0/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e96f673-1cf6-48f1-a55d-347662c75b16/volumes/kubernetes.io~projected/kube-api-access-qvmzm:{mountpoint:/var/lib/kubelet/pods/1e96f673-1cf6-48f1-a55d-347662c75b16/volumes/kubernetes.io~projected/kube-api-access-qvmzm major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22cd77ec-a3f8-4e80-8ffb-def069d70e34/volumes/kubernetes.io~projected/kube-api-access-qg6gq:{mountpoint:/var/lib/kubelet/pods/22cd77ec-a3f8-4e80-8ffb-def069d70e34/volumes/kubernetes.io~projected/kube-api-access-qg6gq major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/283c8ed3-38ac-4a45-82d0-ec5bd9245376/volumes/kubernetes.io~projected/kube-api-access-ngldn:{mountpoint:/var/lib/kubelet/pods/283c8ed3-38ac-4a45-82d0-ec5bd9245376/volumes/kubernetes.io~projected/kube-api-access-ngldn major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c868547-5957-44d5-99d3-4aefc664abb1/volumes/kubernetes.io~projected/kube-api-access-7krck:{mountpoint:/var/lib/kubelet/pods/2c868547-5957-44d5-99d3-4aefc664abb1/volumes/kubernetes.io~projected/kube-api-access-7krck major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d6b618-cc4e-48b8-a7b4-9649a4242aad/volumes/kubernetes.io~projected/kube-api-access-bgpnn:{mountpoint:/var/lib/kubelet/pods/30d6b618-cc4e-48b8-a7b4-9649a4242aad/volumes/kubernetes.io~projected/kube-api-access-bgpnn major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39e68d28-771c-48e8-ba38-f8aad80a6bf9/volumes/kubernetes.io~projected/kube-api-access-jf87f:{mountpoint:/var/lib/kubelet/pods/39e68d28-771c-48e8-ba38-f8aad80a6bf9/volumes/kubernetes.io~projected/kube-api-access-jf87f major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d75d87e-395e-4668-8b7a-42bdd79d08d9/volumes/kubernetes.io~projected/kube-api-access-ltqcp:{mountpoint:/var/lib/kubelet/pods/4d75d87e-395e-4668-8b7a-42bdd79d08d9/volumes/kubernetes.io~projected/kube-api-access-ltqcp major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5013aa7f-b2fd-4df0-b5e3-3478bef510ee/volumes/kubernetes.io~projected/kube-api-access-kzbsq:{mountpoint:/var/lib/kubelet/pods/5013aa7f-b2fd-4df0-b5e3-3478bef510ee/volumes/kubernetes.io~projected/kube-api-access-kzbsq major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64f527e7-dcc8-49be-8c15-a078a7aed186/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/64f527e7-dcc8-49be-8c15-a078a7aed186/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64f527e7-dcc8-49be-8c15-a078a7aed186/volumes/kubernetes.io~projected/kube-api-access-gw5x4:{mountpoint:/var/lib/kubelet/pods/64f527e7-dcc8-49be-8c15-a078a7aed186/volumes/kubernetes.io~projected/kube-api-access-gw5x4 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64f527e7-dcc8-49be-8c15-a078a7aed186/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/64f527e7-dcc8-49be-8c15-a078a7aed186/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68a10036-5c78-4b01-b06d-075a727a5d0b/volumes/kubernetes.io~projected/kube-api-access-s7nbm:{mountpoint:/var/lib/kubelet/pods/68a10036-5c78-4b01-b06d-075a727a5d0b/volumes/kubernetes.io~projected/kube-api-access-s7nbm major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fea26b0-2cef-4683-a20f-620ef0880bfc/volumes/kubernetes.io~projected/kube-api-access-ll47d:{mountpoint:/var/lib/kubelet/pods/6fea26b0-2cef-4683-a20f-620ef0880bfc/volumes/kubernetes.io~projected/kube-api-access-ll47d major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/896fb858-68d1-462c-8a00-2b2b6b5a01af/volumes/kubernetes.io~projected/kube-api-access-6mk86:{mountpoint:/var/lib/kubelet/pods/896fb858-68d1-462c-8a00-2b2b6b5a01af/volumes/kubernetes.io~projected/kube-api-access-6mk86 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/896fb858-68d1-462c-8a00-2b2b6b5a01af/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/896fb858-68d1-462c-8a00-2b2b6b5a01af/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed/volumes/kubernetes.io~projected/kube-api-access-gpz59:{mountpoint:/var/lib/kubelet/pods/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed/volumes/kubernetes.io~projected/kube-api-access-gpz59 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99e51c7f-420e-4709-b098-2ac9f0eaaedb/volumes/kubernetes.io~projected/kube-api-access-cx5xx:{mountpoint:/var/lib/kubelet/pods/99e51c7f-420e-4709-b098-2ac9f0eaaedb/volumes/kubernetes.io~projected/kube-api-access-cx5xx major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99e51c7f-420e-4709-b098-2ac9f0eaaedb/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/99e51c7f-420e-4709-b098-2ac9f0eaaedb/volumes/kubernetes.io~secret/cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cb81e57-c7dc-4447-959f-d5496ffee976/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9cb81e57-c7dc-4447-959f-d5496ffee976/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cb81e57-c7dc-4447-959f-d5496ffee976/volumes/kubernetes.io~projected/kube-api-access-xpqp8:{mountpoint:/var/lib/kubelet/pods/9cb81e57-c7dc-4447-959f-d5496ffee976/volumes/kubernetes.io~projected/kube-api-access-xpqp8 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cb81e57-c7dc-4447-959f-d5496ffee976/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9cb81e57-c7dc-4447-959f-d5496ffee976/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cb81e57-c7dc-4447-959f-d5496ffee976/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9cb81e57-c7dc-4447-959f-d5496ffee976/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fec747e-0efa-40f5-afaf-d45d820700b1/volumes/kubernetes.io~projected/kube-api-access-wcq6w:{mountpoint:/var/lib/kubelet/pods/9fec747e-0efa-40f5-afaf-d45d820700b1/volumes/kubernetes.io~projected/kube-api-access-wcq6w major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a138b9a8-1f14-42a3-97f6-9e2612f83470/volumes/kubernetes.io~projected/kube-api-access-x5mrg:{mountpoint:/var/lib/kubelet/pods/a138b9a8-1f14-42a3-97f6-9e2612f83470/volumes/kubernetes.io~projected/kube-api-access-x5mrg major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6fea55b-74c2-40b1-9952-cd767e448317/volumes/kubernetes.io~projected/kube-api-access-f2xgj:{mountpoint:/var/lib/kubelet/pods/a6fea55b-74c2-40b1-9952-cd767e448317/volumes/kubernetes.io~projected/kube-api-access-f2xgj major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6fea55b-74c2-40b1-9952-cd767e448317/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a6fea55b-74c2-40b1-9952-cd767e448317/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aef015e0-b0a9-4c9f-8140-dd74d8265e64/volumes/kubernetes.io~projected/kube-api-access-tfwzl:{mountpoint:/var/lib/kubelet/pods/aef015e0-b0a9-4c9f-8140-dd74d8265e64/volumes/kubernetes.io~projected/kube-api-access-tfwzl major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afe07c4f-7282-4239-9761-c61ef4153cd9/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/afe07c4f-7282-4239-9761-c61ef4153cd9/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b51723af-6a33-4535-91c0-905ce2a51b24/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b51723af-6a33-4535-91c0-905ce2a51b24/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcc543bf-8ffd-488d-999a-2b8da64e9c31/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/bcc543bf-8ffd-488d-999a-2b8da64e9c31/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcc543bf-8ffd-488d-999a-2b8da64e9c31/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/bcc543bf-8ffd-488d-999a-2b8da64e9c31/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcc543bf-8ffd-488d-999a-2b8da64e9c31/volumes/kubernetes.io~projected/kube-api-access-snn5k:{mountpoint:/var/lib/kubelet/pods/bcc543bf-8ffd-488d-999a-2b8da64e9c31/volumes/kubernetes.io~projected/kube-api-access-snn5k major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfe84946-82e9-456a-b81f-af887093749b/volumes/kubernetes.io~projected/kube-api-access-b6tkr:{mountpoint:/var/lib/kubelet/pods/bfe84946-82e9-456a-b81f-af887093749b/volumes/kubernetes.io~projected/kube-api-access-b6tkr major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd/volumes/kubernetes.io~projected/kube-api-access-hd99v:{mountpoint:/var/lib/kubelet/pods/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd/volumes/kubernetes.io~projected/kube-api-access-hd99v major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb6295a6-f31e-48a3-b4df-7d872a961161/volumes/kubernetes.io~projected/kube-api-access-v8mlg:{mountpoint:/var/lib/kubelet/pods/cb6295a6-f31e-48a3-b4df-7d872a961161/volumes/kubernetes.io~projected/kube-api-access-v8mlg major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb6295a6-f31e-48a3-b4df-7d872a961161/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/cb6295a6-f31e-48a3-b4df-7d872a961161/volumes/kubernetes.io~secret/default-certificate major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb6295a6-f31e-48a3-b4df-7d872a961161/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/cb6295a6-f31e-48a3-b4df-7d872a961161/volumes/kubernetes.io~secret/stats-auth major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d23cf40e-53e9-406e-92f8-ce628b1baa68/volumes/kubernetes.io~projected/kube-api-access-d8r65:{mountpoint:/var/lib/kubelet/pods/d23cf40e-53e9-406e-92f8-ce628b1baa68/volumes/kubernetes.io~projected/kube-api-access-d8r65 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d23cf40e-53e9-406e-92f8-ce628b1baa68/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d23cf40e-53e9-406e-92f8-ce628b1baa68/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb120384-9c46-4dbd-bb80-bedf4bbe957c/volumes/kubernetes.io~projected/kube-api-access-5nczq:{mountpoint:/var/lib/kubelet/pods/eb120384-9c46-4dbd-bb80-bedf4bbe957c/volumes/kubernetes.io~projected/kube-api-access-5nczq major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb120384-9c46-4dbd-bb80-bedf4bbe957c/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/eb120384-9c46-4dbd-bb80-bedf4bbe957c/volumes/kubernetes.io~secret/signing-key major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f22a4aac-bd75-41d2-b1c2-9293bb405389/volumes/kubernetes.io~projected/kube-api-access-2fknt:{mountpoint:/var/lib/kubelet/pods/f22a4aac-bd75-41d2-b1c2-9293bb405389/volumes/kubernetes.io~projected/kube-api-access-2fknt major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f65dc226-70a9-4fcf-ae3b-28559c2ff235/volumes/kubernetes.io~projected/kube-api-access-8bskm:{mountpoint:/var/lib/kubelet/pods/f65dc226-70a9-4fcf-ae3b-28559c2ff235/volumes/kubernetes.io~projected/kube-api-access-8bskm major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f65dc226-70a9-4fcf-ae3b-28559c2ff235/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f65dc226-70a9-4fcf-ae3b-28559c2ff235/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8dd1189-9c04-4ba1-a349-04e7dabcab52/volumes/kubernetes.io~projected/kube-api-access-t22cd:{mountpoint:/var/lib/kubelet/pods/f8dd1189-9c04-4ba1-a349-04e7dabcab52/volumes/kubernetes.io~projected/kube-api-access-t22cd major:0 minor:287 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/38beb775ff39c653339261f5935bde7590b5b96d1bc8ff29a9a1d31ef8df63a1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/7db585b650fa0a3d79a876156c48b7deaf7b49a795fe53e758fb6d1e4ec45fb8/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f6259228459778c0a788339bdaa67d84cc0a33db3318e321f41c0697567c0156/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/f6acbf02284a9c94588a9c2069017f656bcf3b23792c084da70dc375cc7e8c3a/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/2442706c1930e26a4edbbfc08fdcdf955c37a1de6992930bb071a0efe68abbb8/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/27f69fcac51b3a5119d1df6ff740e385c86f4975083bc4969aa0ceba1e0717e3/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/d4b05fd3427da4e6082f967ed2a9ed5d8b962afcf31176c3c2f0c7795d0c4a24/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/854f5564ba5d9c59574dbc0d9a1840b99d814e4dd2041d1effd3c30f24e66478/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/eeff33338a498ce0fe2b25edabb73401b808a84a8e317e55817a6e01317c4f8d/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/0e7928d75d0b1f58e23174978cbdecc2270310aee6bad59fc1c8adaa8e85ca9b/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/b21e9d8a6d818f1a3618a49b1655ad901135fb831363169a31344fa50a8371f0/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/22a32618f3dda1ee3a0c9959164ae89294d4583f1b923d7e4cf0c56ff5afe021/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/694b721e3fae82e3676f51ca78df6f967d5d3b57e47a253bcc24aa7155067e35/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/6d48742a553e5f9ee493aabb98380e5bdf1b11e4a4679285148f2c4260324fe3/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/498c7e8484bbedd23e2fdd97072de664c604f5bc8e78509bbc472840b4061043/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/09ee87f5534193b75085b0cce167dffae770829feaa9da2a2d764f18587541c5/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/cd16e9de018acfcf8c3a1b98fb6a329c61bc2c66ccc080f3e57f4db3faf92782/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/cb756a1bef3bc1a13bb35628a07f3eeabad28a1e9c13c5e820b2c0ea56120b92/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/84ca644f40a67393ffbe81eac3cb93871f1da4b93b4750d79a64b9b028300df9/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b97fc1a30799b980e222dbe99f0bf909b1bd09b37933b2dcfb5fe001abf75c73/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-188:{mountpoint:/var/lib/containers/storage/overlay/d3f2d95467b1d2dbfd0a971984a376e36641ce71ffebb8605d4e2cfcf0939f9f/merged major:0 minor:188 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/bb05fc0563c705d11a2819a8a06795e7244f4aa5a04d359688f6c08b1b0c41c5/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/b04a2822868919b48adb68e3eb372ddfd5979b55c455f69f3c39bb94af2866f5/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/7fc3092463b5ee2ee04c930e3dc1224e50896b93ff2c922fa5f1f447df4dc67f/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/9c5dcff95d7eb7931b19da256878e93d37cf6171515818c2ee24129e70a71c8f/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/9a40d50e4dd11c7a2a8967dd8bd1c55b9c101a1aa37f8999d1e5a222d2b5a41a/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/130552410b0a998207c8aa97b08516b9c7619294641d4a96a8ae2d62429a62e9/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/f9a9a11997a8d296180e646da88329bfe87b9dd6ac96a796409abc19fcb6c468/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/0fbaa713f6e40bae96dffd4698b48db9f1d731c6de0f79e4e42412d8147f4cdf/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/2eb330a62f453eb5b46f23a26f6b9d41b7cc739c172509c6c174780681010ef1/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/20648d842a1a0bfdd793e98e950ceb7d13f386f4cc2e5b1d3bba817876ffea7c/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/a58060ccb68979d653eb3a60f5f12514d0d75c88bb8766f2c8826dc8e1d2d6a7/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/56124b413d31c4f013e8f3e738e2d9836657737db4622800ff9f137d3c3008a0/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/51b51cd6be06ec282652568b94b1a62da24347954c5534ce583edf0d08c40680/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/7804768b26b7644a047bcd8cec74eb3a3841e3a952f5c7a4783d5d7f99986582/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/e3547bbcd4f255389a222cf91de694cec3d8990622629a2752c594d11eb0419c/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/5ac2bd6fc4ef4f12cc2cfa5743a2fda045bf422f6d57194f553b20dc2437c0b0/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/9f01509f166b0af4221225ed18f1bc6762be5373e3b5a3c790e5e4be554603bd/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/80c502234ad678a684dcd57f5f8fddc8d5083ef9aa79ff13bde395a4dc34827d/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/cfdf4fc34dc36e64d47597112f37bf12f76c1de9cd2052afec5f77b23b9ff065/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/bc00268273ca1879f03d2213002a32b99020fd8682467eebae22196c3422a044/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/a397db3303b1401be06493bffe48bdbd48aad07de38ed65b97ea8b639ced4df2/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/e5e60147d9b1e1c6b9a335457d02d65248aadb08246451b0e4981f33b6a52661/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/19e0a3696045688bcf5ffec8c0fc54fe2e0a957946a5477f129916469ba186ff/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/217b0ed0e94abcaa0ce2a77fe3ace9454d1445d17d89629b6020c9e0119e0805/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/1990aeafeda2a202f9b15242f87510b50d638780bbbb6b304211bb2d45739658/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/3334f13874efef2fbb685b1a2dec6198bf30d84bf0fb382112b1ae603f04aba8/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/207e90a43594d6850327391c58e9ff5fbc391281eec2c5b1500ab93179caf942/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/73fa205382eebe0b293e7bd7f76433b6b232a9130abd000e99f0294323b593bf/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/1a441f562782679415f53dc7472d70c4e274504ca137e148d03980d22f2d46a8/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/b4701e89e9c6bf4890b26861760dc944fb2f1db6b279df4df13623e1f342234e/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/e43c758a7df223e210851d87fd4de1243b0d267149350add7c64e4bc6c4a9d56/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/15e93b7b6e0ae46a40d62f2c3179043646b3ecc304fbdda85a8c2c6c00f89d0b/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/6f108bad9655a1f54b761287f48193c7e00bae996394808cd63f4c2d3a3056a0/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/41223f4d29887ea2e95c22778e1535a56fc014e791aeeea628536277e042201a/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/f1f43bf6a8cd34dd4f26e901eb7443952a0006e9760a5bfed4d32c4192aeaae4/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e3f1f5cfaac0720fb9426ddf8695aacd4ae6963a4a0173d7e9c620b1895957f5/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/40cef23c01bb2e00abe39a30a4101ad36f7154e17bd7945146ddc99e4b251d1b/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 20:14:31.808194 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.807839 4971 manager.go:217] Machine: {Timestamp:2026-11-02 20:14:31.807391135 +0000 UTC m=+0.097047660 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101132 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29165ebcaa2f561399757821b384f2 SystemUUID:ec29165e-bcaa-2f56-1399-757821b384f2 BootID:a53b8b62-6e60-4487-a1c5-941309010acc Filesystems:[{Device:/run/containers/storage/overlay-containers/bc66d97e073c8610d57e2e9b89438cf1dd411d9b8cfa417c158b770ee2cdf1cf/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6fea26b0-2cef-4683-a20f-620ef0880bfc/volumes/kubernetes.io~projected/kube-api-access-ll47d DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee3b20c44102840b9bf245e66c89b2e0941ff66d2bcd0140c2f3d052154e3f71/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c7aeeaa257f53ea9aa8d328684a6c694a927c24739b9dbd19481ec1d8ec2d30/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-188 DeviceMajor:0 DeviceMinor:188 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8dd1189-9c04-4ba1-a349-04e7dabcab52/volumes/kubernetes.io~projected/kube-api-access-t22cd DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64f527e7-dcc8-49be-8c15-a078a7aed186/volumes/kubernetes.io~projected/kube-api-access-gw5x4 DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07ab293df918258cd3b264ce3c4eedc9958d2dae6b334a34d3a6e19c5c41af80/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb6295a6-f31e-48a3-b4df-7d872a961161/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cb6295a6-f31e-48a3-b4df-7d872a961161/volumes/kubernetes.io~projected/kube-api-access-v8mlg DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7873bc74331e14113791f9d5c98f8803f2300f77f1296ca2975e178b121fb289/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f22a4aac-bd75-41d2-b1c2-9293bb405389/volumes/kubernetes.io~projected/kube-api-access-2fknt DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64f527e7-dcc8-49be-8c15-a078a7aed186/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bcc543bf-8ffd-488d-999a-2b8da64e9c31/volumes/kubernetes.io~projected/kube-api-access-snn5k DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cb81e57-c7dc-4447-959f-d5496ffee976/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/896fb858-68d1-462c-8a00-2b2b6b5a01af/volumes/kubernetes.io~projected/kube-api-access-6mk86 DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9cb81e57-c7dc-4447-959f-d5496ffee976/volumes/kubernetes.io~projected/kube-api-access-xpqp8 DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/636ed99fad21ca95ddac9d9ea7049bbaec0c1cae39d728cbf0aff5d11b0dd426/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15606ef9514cdd2b5c88060ac28788605d80266d1a0ec4a2b215b17077900d8d/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5013aa7f-b2fd-4df0-b5e3-3478bef510ee/volumes/kubernetes.io~projected/kube-api-access-kzbsq DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99e51c7f-420e-4709-b098-2ac9f0eaaedb/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6151eb6e612794d3cdae41786b68fae9972d326cb1b3e497e02d5d72f9bf66f/userdata/shm DeviceMajor:0 DeviceMinor:322 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/68a10036-5c78-4b01-b06d-075a727a5d0b/volumes/kubernetes.io~projected/kube-api-access-s7nbm DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6fea55b-74c2-40b1-9952-cd767e448317/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f34649d86715069ceade5514d1e1fc853e5db5d7b431b9df51b5d232244de298/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6cc565920073e6b826eb68c89265880f87092b390ddbfaf0f49c582b108cb60/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fbcf8209bfc27d3f3e1b23824d5dd3abfe29021d719ae75cb9fa223d7f3297dd/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/896fb858-68d1-462c-8a00-2b2b6b5a01af/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2c868547-5957-44d5-99d3-4aefc664abb1/volumes/kubernetes.io~projected/kube-api-access-7krck DeviceMajor:0 DeviceMinor:312 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f65dc226-70a9-4fcf-ae3b-28559c2ff235/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bfe84946-82e9-456a-b81f-af887093749b/volumes/kubernetes.io~projected/kube-api-access-b6tkr DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9fec747e-0efa-40f5-afaf-d45d820700b1/volumes/kubernetes.io~projected/kube-api-access-wcq6w DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14e005f90d52a2dc6b1bcb210c65223609ecf4699f6910ad0cf71b52d3e38265/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/283c8ed3-38ac-4a45-82d0-ec5bd9245376/volumes/kubernetes.io~projected/kube-api-access-ngldn DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/02eded99-edf7-41a0-8866-549252be5fe0/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/02eded99-edf7-41a0-8866-549252be5fe0/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f614f8ecc585a5a2ded495c5f6f82ce305420b1a56eba73a519e203e8c03310/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/b51723af-6a33-4535-91c0-905ce2a51b24/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/612faca9a4be33f531f8cf9d8762516888219f40744f562cdb9035696360431c/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa2af273ee991d426f21609d08d24b63d88a5cd6378432b8dc7f8ff622fd7937/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/afe07c4f-7282-4239-9761-c61ef4153cd9/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca238d1beee6588afe58ce0417b419e77f5fc5f70fda1e92edaef7c4d552b662/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39e68d28-771c-48e8-ba38-f8aad80a6bf9/volumes/kubernetes.io~projected/kube-api-access-jf87f DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64f527e7-dcc8-49be-8c15-a078a7aed186/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/22cd77ec-a3f8-4e80-8ffb-def069d70e34/volumes/kubernetes.io~projected/kube-api-access-qg6gq DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82c2cb92ed201c94477cd9681fb845f1e683c68a36351893b5b706f136e061c1/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/690a5c20ac6d5a4dc1bf8a9ccf0448d9b7d97f3c5d154b65e2b20a6534b579e6/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd/volumes/kubernetes.io~projected/kube-api-access-hd99v DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb120384-9c46-4dbd-bb80-bedf4bbe957c/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:342 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e96f673-1cf6-48f1-a55d-347662c75b16/volumes/kubernetes.io~projected/kube-api-access-qvmzm DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7bf224f0387a6c1763335ce89ce4f008842b70867aa913d224958400e5e831fe/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/02eded99-edf7-41a0-8866-549252be5fe0/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed/volumes/kubernetes.io~projected/kube-api-access-gpz59 DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/eb120384-9c46-4dbd-bb80-bedf4bbe957c/volumes/kubernetes.io~projected/kube-api-access-5nczq DeviceMajor:0 DeviceMinor:343 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bcc543bf-8ffd-488d-999a-2b8da64e9c31/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbc0f8137d19d2acb1fda8e0914da1d28884aabc4d694a37da80cd622155e926/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6fea55b-74c2-40b1-9952-cd767e448317/volumes/kubernetes.io~projected/kube-api-access-f2xgj DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4d75d87e-395e-4668-8b7a-42bdd79d08d9/volumes/kubernetes.io~projected/kube-api-access-ltqcp DeviceMajor:0 DeviceMinor:317 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71c7ced32f7d40df0fdd5e71f6234eda39d1c2a35f033827b6472f75ccd11856/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd0d3d1b1070f9fd7a896f2b22158a6cc02853c2bc49691f7dbd0c1fda88b9cc/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d23cf40e-53e9-406e-92f8-ce628b1baa68/volumes/kubernetes.io~projected/kube-api-access-d8r65 DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f65dc226-70a9-4fcf-ae3b-28559c2ff235/volumes/kubernetes.io~projected/kube-api-access-8bskm DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af81f493cee046ae85b7be06bade3ce4506994eba8b5dda4076bf6413296b13b/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ea136b3030a2d7db3d157c0d72d2877085de68f36077b113c39ef0a8d689e27/userdata/shm DeviceMajor:0 DeviceMinor:52 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d23cf40e-53e9-406e-92f8-ce628b1baa68/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cb6295a6-f31e-48a3-b4df-7d872a961161/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/aef015e0-b0a9-4c9f-8140-dd74d8265e64/volumes/kubernetes.io~projected/kube-api-access-tfwzl DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9cb81e57-c7dc-4447-959f-d5496ffee976/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:221 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bcc543bf-8ffd-488d-999a-2b8da64e9c31/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/30d6b618-cc4e-48b8-a7b4-9649a4242aad/volumes/kubernetes.io~projected/kube-api-access-bgpnn DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/99e51c7f-420e-4709-b098-2ac9f0eaaedb/volumes/kubernetes.io~projected/kube-api-access-cx5xx DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a138b9a8-1f14-42a3-97f6-9e2612f83470/volumes/kubernetes.io~projected/kube-api-access-x5mrg DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9cb81e57-c7dc-4447-959f-d5496ffee976/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/02eded99-edf7-41a0-8866-549252be5fe0/volumes/kubernetes.io~projected/kube-api-access-kmsvc DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:636ed99fad21ca9 MacAddress:b6:08:97:98:0c:9e Speed:10000 Mtu:8901} {Name:71c7ced32f7d40d MacAddress:a2:0c:20:17:59:b6 Speed:10000 Mtu:8901} {Name:7873bc74331e141 MacAddress:56:13:19:47:07:26 Speed:10000 Mtu:8901} {Name:82c2cb92ed201c9 MacAddress:82:3f:1d:27:3d:3a Speed:10000 Mtu:8901} {Name:9f614f8ecc585a5 MacAddress:d6:23:2e:7f:78:9e Speed:10000 Mtu:8901} {Name:aa2af273ee991d4 MacAddress:a2:29:b7:c3:46:02 Speed:10000 Mtu:8901} {Name:af81f493cee046a MacAddress:4e:47:b5:c3:e7:49 Speed:10000 Mtu:8901} {Name:b6cc565920073e6 MacAddress:5a:4b:05:f2:bb:f3 Speed:10000 Mtu:8901} {Name:bd0d3d1b1070f9f MacAddress:5e:22:14:17:1a:ce Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c1:17:f7:41 Speed:0 Mtu:9001} {Name:br-int MacAddress:72:c2:27:a7:44:55 Speed:0 Mtu:8901} {Name:ca238d1beee6588 MacAddress:1a:61:40:35:78:2e Speed:10000 Mtu:8901} {Name:d6151eb6e612794 MacAddress:a2:25:1a:79:20:97 Speed:10000 Mtu:8901} {Name:ee3b20c44102840 MacAddress:22:28:5f:fe:43:f0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:c1:17:f7:41 Speed:0 Mtu:9001} {Name:f34649d86715069 MacAddress:fa:10:b6:af:4c:55 Speed:10000 Mtu:8901} {Name:fbcf8209bfc27d3 MacAddress:2a:e6:4e:5a:ae:b9 Speed:10000 Mtu:8901} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1e:fa:0f:c3:bb:78 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 20:14:31.808449 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.808199 4971 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 20:14:31.808449 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.808240 4971 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 20:14:31.808583 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.808561 4971 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 20:14:31.808739 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.808586 4971 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-120.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 20:14:31.808813 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.808742 4971 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 20:14:31.808813 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.808755 4971 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 20:14:31.808813 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.808769 4971 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 20:14:31.808813 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.808787 4971 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 20:14:31.808813 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.808812 4971 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 20:14:31.809038 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.809024 4971 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 20:14:31.809179 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.809168 4971 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 20:14:31.809236 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.809213 4971 kubelet.go:493] "Attempting to sync node with API server" Nov 02 20:14:31.809236 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.809227 4971 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 20:14:31.809334 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.809242 4971 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 20:14:31.809334 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.809255 4971 kubelet.go:405] "Adding apiserver pod source" Nov 02 20:14:31.809334 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.809266 4971 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 20:14:31.809944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.809929 4971 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 20:14:31.809944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.809942 4971 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 20:14:31.810508 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.810486 4971 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 20:14:31.813752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.813722 4971 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 20:14:31.813752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.813738 4971 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 20:14:31.813902 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.813837 4971 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 20:14:31.814030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.814016 4971 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 20:14:31.814094 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.814046 4971 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 20:14:31.814186 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.814175 4971 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 20:14:31.814232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.814191 4971 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 20:14:31.814232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.814198 4971 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 20:14:31.814232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.814205 4971 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 20:14:31.814232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.814213 4971 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 20:14:31.814232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.814222 4971 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 20:14:31.814232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.814232 4971 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 20:14:31.814515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.814240 4971 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 20:14:31.814515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.814247 4971 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 20:14:31.814515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.814256 4971 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 20:14:31.814515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.814268 4971 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 20:14:31.814515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.814282 4971 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 20:14:31.814515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.814316 4971 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 20:14:31.814515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.814326 4971 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 20:14:31.818220 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.818206 4971 server.go:1263] "Started kubelet" Nov 02 20:14:31.818361 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.818305 4971 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 20:14:31.818626 ip-10-0-140-120 systemd[1]: Started Kubernetes Kubelet. Nov 02 20:14:31.818740 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.818689 4971 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 20:14:31.818800 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.818763 4971 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 20:14:31.819199 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.819051 4971 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 20:14:31.819635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.819614 4971 server.go:317] "Adding debug handlers to kubelet server" Nov 02 20:14:31.821442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.821424 4971 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 20:14:31.821899 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.821446 4971 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 20:14:31.821899 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.821499 4971 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 20:08:51 +0000 UTC" deadline="2028-05-02 17:12:53.958257429 +0000 UTC" Nov 02 20:14:31.821899 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.821530 4971 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13124h58m22.136730247s" Nov 02 20:14:31.821899 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.821560 4971 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 20:14:31.821899 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.821566 4971 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 20:14:31.821899 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:31.821571 4971 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-120.us-west-2.compute.internal\" not found" Nov 02 20:14:31.821899 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.821653 4971 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.823467 4971 factory.go:55] Registering systemd factory Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.823488 4971 factory.go:223] Registration of the systemd container factory successfully Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.823916 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d75d87e-395e-4668-8b7a-42bdd79d08d9" volumeName="kubernetes.io/projected/4d75d87e-395e-4668-8b7a-42bdd79d08d9-kube-api-access-ltqcp" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.823938 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64f527e7-dcc8-49be-8c15-a078a7aed186" volumeName="kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-ovnkube-script-lib" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.823954 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb6295a6-f31e-48a3-b4df-7d872a961161" volumeName="kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-stats-auth" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.823969 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb120384-9c46-4dbd-bb80-bedf4bbe957c" volumeName="kubernetes.io/configmap/eb120384-9c46-4dbd-bb80-bedf4bbe957c-signing-cabundle" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.823983 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02eded99-edf7-41a0-8866-549252be5fe0" volumeName="kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-kube-api-access-kmsvc" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.823994 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="896fb858-68d1-462c-8a00-2b2b6b5a01af" volumeName="kubernetes.io/configmap/896fb858-68d1-462c-8a00-2b2b6b5a01af-config" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824005 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed" volumeName="kubernetes.io/projected/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-kube-api-access-gpz59" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824016 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fec747e-0efa-40f5-afaf-d45d820700b1" volumeName="kubernetes.io/projected/9fec747e-0efa-40f5-afaf-d45d820700b1-kube-api-access-wcq6w" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824030 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8dd1189-9c04-4ba1-a349-04e7dabcab52" volumeName="kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824042 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02eded99-edf7-41a0-8866-549252be5fe0" volumeName="kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-image-registry-private-configuration" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824053 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68a10036-5c78-4b01-b06d-075a727a5d0b" volumeName="kubernetes.io/configmap/68a10036-5c78-4b01-b06d-075a727a5d0b-iptables-alerter-script" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824063 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afe07c4f-7282-4239-9761-c61ef4153cd9" volumeName="kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824120 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b51723af-6a33-4535-91c0-905ce2a51b24" volumeName="kubernetes.io/secret/b51723af-6a33-4535-91c0-905ce2a51b24-agent-certs" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824137 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d6b618-cc4e-48b8-a7b4-9649a4242aad" volumeName="kubernetes.io/projected/30d6b618-cc4e-48b8-a7b4-9649a4242aad-kube-api-access-bgpnn" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824151 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39e68d28-771c-48e8-ba38-f8aad80a6bf9" volumeName="kubernetes.io/projected/39e68d28-771c-48e8-ba38-f8aad80a6bf9-kube-api-access-jf87f" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824180 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a138b9a8-1f14-42a3-97f6-9e2612f83470" volumeName="kubernetes.io/projected/a138b9a8-1f14-42a3-97f6-9e2612f83470-kube-api-access-x5mrg" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824208 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcc543bf-8ffd-488d-999a-2b8da64e9c31" volumeName="kubernetes.io/empty-dir/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-tuned" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824243 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="283c8ed3-38ac-4a45-82d0-ec5bd9245376" volumeName="kubernetes.io/configmap/283c8ed3-38ac-4a45-82d0-ec5bd9245376-config-volume" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824366 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5013aa7f-b2fd-4df0-b5e3-3478bef510ee" volumeName="kubernetes.io/projected/5013aa7f-b2fd-4df0-b5e3-3478bef510ee-kube-api-access-kzbsq" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824454 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfe84946-82e9-456a-b81f-af887093749b" volumeName="kubernetes.io/projected/bfe84946-82e9-456a-b81f-af887093749b-kube-api-access-b6tkr" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824472 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb6295a6-f31e-48a3-b4df-7d872a961161" volumeName="kubernetes.io/projected/cb6295a6-f31e-48a3-b4df-7d872a961161-kube-api-access-v8mlg" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824487 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb81e57-c7dc-4447-959f-d5496ffee976" volumeName="kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-bound-sa-token" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824515 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd" volumeName="kubernetes.io/configmap/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-serviceca" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824533 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f65dc226-70a9-4fcf-ae3b-28559c2ff235" volumeName="kubernetes.io/projected/f65dc226-70a9-4fcf-ae3b-28559c2ff235-kube-api-access-8bskm" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824565 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e96f673-1cf6-48f1-a55d-347662c75b16" volumeName="kubernetes.io/projected/1e96f673-1cf6-48f1-a55d-347662c75b16-kube-api-access-qvmzm" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824597 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64f527e7-dcc8-49be-8c15-a078a7aed186" volumeName="kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-ovnkube-config" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824614 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fea26b0-2cef-4683-a20f-620ef0880bfc" volumeName="kubernetes.io/configmap/6fea26b0-2cef-4683-a20f-620ef0880bfc-telemetry-config" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824670 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb81e57-c7dc-4447-959f-d5496ffee976" volumeName="kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-image-registry-private-configuration" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824696 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fec747e-0efa-40f5-afaf-d45d820700b1" volumeName="kubernetes.io/configmap/9fec747e-0efa-40f5-afaf-d45d820700b1-cni-binary-copy" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824714 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f22a4aac-bd75-41d2-b1c2-9293bb405389" volumeName="kubernetes.io/projected/f22a4aac-bd75-41d2-b1c2-9293bb405389-kube-api-access-2fknt" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824751 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02eded99-edf7-41a0-8866-549252be5fe0" volumeName="kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-installation-pull-secrets" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824787 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64f527e7-dcc8-49be-8c15-a078a7aed186" volumeName="kubernetes.io/projected/64f527e7-dcc8-49be-8c15-a078a7aed186-kube-api-access-gw5x4" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824802 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="896fb858-68d1-462c-8a00-2b2b6b5a01af" volumeName="kubernetes.io/projected/896fb858-68d1-462c-8a00-2b2b6b5a01af-kube-api-access-6mk86" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824844 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed" volumeName="kubernetes.io/configmap/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-config" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824864 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99e51c7f-420e-4709-b098-2ac9f0eaaedb" volumeName="kubernetes.io/secret/99e51c7f-420e-4709-b098-2ac9f0eaaedb-cert" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824901 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a138b9a8-1f14-42a3-97f6-9e2612f83470" volumeName="kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-cni-binary-copy" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824955 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6fea55b-74c2-40b1-9952-cd767e448317" volumeName="kubernetes.io/configmap/a6fea55b-74c2-40b1-9952-cd767e448317-trusted-ca" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824971 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b51723af-6a33-4535-91c0-905ce2a51b24" volumeName="kubernetes.io/configmap/b51723af-6a33-4535-91c0-905ce2a51b24-konnectivity-ca" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.824987 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d75d87e-395e-4668-8b7a-42bdd79d08d9" volumeName="kubernetes.io/empty-dir/4d75d87e-395e-4668-8b7a-42bdd79d08d9-archives-path" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825046 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68a10036-5c78-4b01-b06d-075a727a5d0b" volumeName="kubernetes.io/projected/68a10036-5c78-4b01-b06d-075a727a5d0b-kube-api-access-s7nbm" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825119 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fea26b0-2cef-4683-a20f-620ef0880bfc" volumeName="kubernetes.io/projected/6fea26b0-2cef-4683-a20f-620ef0880bfc-kube-api-access-ll47d" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825145 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99e51c7f-420e-4709-b098-2ac9f0eaaedb" volumeName="kubernetes.io/projected/99e51c7f-420e-4709-b098-2ac9f0eaaedb-kube-api-access-cx5xx" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825173 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d23cf40e-53e9-406e-92f8-ce628b1baa68" volumeName="kubernetes.io/projected/d23cf40e-53e9-406e-92f8-ce628b1baa68-kube-api-access-d8r65" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825190 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f65dc226-70a9-4fcf-ae3b-28559c2ff235" volumeName="kubernetes.io/configmap/f65dc226-70a9-4fcf-ae3b-28559c2ff235-service-ca-bundle" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825205 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02eded99-edf7-41a0-8866-549252be5fe0" volumeName="kubernetes.io/empty-dir/02eded99-edf7-41a0-8866-549252be5fe0-ca-trust-extracted" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825231 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02eded99-edf7-41a0-8866-549252be5fe0" volumeName="kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-bound-sa-token" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825246 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="283c8ed3-38ac-4a45-82d0-ec5bd9245376" volumeName="kubernetes.io/empty-dir/283c8ed3-38ac-4a45-82d0-ec5bd9245376-tmp-dir" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825270 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d23cf40e-53e9-406e-92f8-ce628b1baa68" volumeName="kubernetes.io/secret/d23cf40e-53e9-406e-92f8-ce628b1baa68-serving-cert" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825299 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb120384-9c46-4dbd-bb80-bedf4bbe957c" volumeName="kubernetes.io/secret/eb120384-9c46-4dbd-bb80-bedf4bbe957c-signing-key" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825327 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64f527e7-dcc8-49be-8c15-a078a7aed186" volumeName="kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-env-overrides" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825341 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcc543bf-8ffd-488d-999a-2b8da64e9c31" volumeName="kubernetes.io/empty-dir/bcc543bf-8ffd-488d-999a-2b8da64e9c31-tmp" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825365 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d23cf40e-53e9-406e-92f8-ce628b1baa68" volumeName="kubernetes.io/configmap/d23cf40e-53e9-406e-92f8-ce628b1baa68-config" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825385 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f65dc226-70a9-4fcf-ae3b-28559c2ff235" volumeName="kubernetes.io/configmap/f65dc226-70a9-4fcf-ae3b-28559c2ff235-trusted-ca-bundle" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825400 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02eded99-edf7-41a0-8866-549252be5fe0" volumeName="kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-trusted-ca" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825429 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22cd77ec-a3f8-4e80-8ffb-def069d70e34" volumeName="kubernetes.io/projected/22cd77ec-a3f8-4e80-8ffb-def069d70e34-kube-api-access-qg6gq" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825489 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="283c8ed3-38ac-4a45-82d0-ec5bd9245376" volumeName="kubernetes.io/projected/283c8ed3-38ac-4a45-82d0-ec5bd9245376-kube-api-access-ngldn" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825517 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39e68d28-771c-48e8-ba38-f8aad80a6bf9" volumeName="kubernetes.io/empty-dir/39e68d28-771c-48e8-ba38-f8aad80a6bf9-tmp-dir" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825557 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb81e57-c7dc-4447-959f-d5496ffee976" volumeName="kubernetes.io/empty-dir/9cb81e57-c7dc-4447-959f-d5496ffee976-ca-trust-extracted" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825571 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d5e73f9-907c-40b1-a596-b079ded3abfb" volumeName="kubernetes.io/configmap/9d5e73f9-907c-40b1-a596-b079ded3abfb-nginx-conf" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825585 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6fea55b-74c2-40b1-9952-cd767e448317" volumeName="kubernetes.io/secret/a6fea55b-74c2-40b1-9952-cd767e448317-serving-cert" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825634 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd" volumeName="kubernetes.io/projected/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-kube-api-access-hd99v" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825659 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02eded99-edf7-41a0-8866-549252be5fe0" volumeName="kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-registry-certificates" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825676 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d75d87e-395e-4668-8b7a-42bdd79d08d9" volumeName="kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-trusted-ca-bundle" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825753 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="896fb858-68d1-462c-8a00-2b2b6b5a01af" volumeName="kubernetes.io/secret/896fb858-68d1-462c-8a00-2b2b6b5a01af-serving-cert" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825799 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb81e57-c7dc-4447-959f-d5496ffee976" volumeName="kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-certificates" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825831 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb81e57-c7dc-4447-959f-d5496ffee976" volumeName="kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-installation-pull-secrets" seLinuxMountContext="" Nov 02 20:14:31.825830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825863 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aef015e0-b0a9-4c9f-8140-dd74d8265e64" volumeName="kubernetes.io/projected/aef015e0-b0a9-4c9f-8140-dd74d8265e64-kube-api-access-tfwzl" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825892 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcc543bf-8ffd-488d-999a-2b8da64e9c31" volumeName="kubernetes.io/projected/bcc543bf-8ffd-488d-999a-2b8da64e9c31-kube-api-access-snn5k" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825905 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb6295a6-f31e-48a3-b4df-7d872a961161" volumeName="kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-default-certificate" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825925 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64f527e7-dcc8-49be-8c15-a078a7aed186" volumeName="kubernetes.io/secret/64f527e7-dcc8-49be-8c15-a078a7aed186-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825939 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb81e57-c7dc-4447-959f-d5496ffee976" volumeName="kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-kube-api-access-xpqp8" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825950 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a138b9a8-1f14-42a3-97f6-9e2612f83470" volumeName="kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825981 4971 factory.go:153] Registering CRI-O factory Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826005 4971 factory.go:223] Registration of the crio container factory successfully Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.825974 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6fea55b-74c2-40b1-9952-cd767e448317" volumeName="kubernetes.io/projected/a6fea55b-74c2-40b1-9952-cd767e448317-kube-api-access-f2xgj" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826074 4971 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826146 4971 factory.go:103] Registering Raw factory Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826163 4971 manager.go:1199] Started watching for new ooms in manager Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826095 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb120384-9c46-4dbd-bb80-bedf4bbe957c" volumeName="kubernetes.io/projected/eb120384-9c46-4dbd-bb80-bedf4bbe957c-kube-api-access-5nczq" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826182 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f65dc226-70a9-4fcf-ae3b-28559c2ff235" volumeName="kubernetes.io/empty-dir/f65dc226-70a9-4fcf-ae3b-28559c2ff235-tmp" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826199 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f65dc226-70a9-4fcf-ae3b-28559c2ff235" volumeName="kubernetes.io/secret/f65dc226-70a9-4fcf-ae3b-28559c2ff235-serving-cert" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826223 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c868547-5957-44d5-99d3-4aefc664abb1" volumeName="kubernetes.io/projected/2c868547-5957-44d5-99d3-4aefc664abb1-kube-api-access-7krck" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826237 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d75d87e-395e-4668-8b7a-42bdd79d08d9" volumeName="kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-service-ca-bundle" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826251 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb81e57-c7dc-4447-959f-d5496ffee976" volumeName="kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-trusted-ca" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826281 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fec747e-0efa-40f5-afaf-d45d820700b1" volumeName="kubernetes.io/configmap/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-daemon-config" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826304 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6fea55b-74c2-40b1-9952-cd767e448317" volumeName="kubernetes.io/configmap/a6fea55b-74c2-40b1-9952-cd767e448317-config" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826315 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f65dc226-70a9-4fcf-ae3b-28559c2ff235" volumeName="kubernetes.io/empty-dir/f65dc226-70a9-4fcf-ae3b-28559c2ff235-snapshots" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826349 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c868547-5957-44d5-99d3-4aefc664abb1" volumeName="kubernetes.io/empty-dir/2c868547-5957-44d5-99d3-4aefc664abb1-data-volume" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826381 4971 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a138b9a8-1f14-42a3-97f6-9e2612f83470" volumeName="kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826394 4971 reconstruct.go:97] "Volume reconstruction finished" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826403 4971 reconciler.go:29] "Reconciler: start to sync state" Nov 02 20:14:31.830147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.826669 4971 manager.go:319] Starting recovery of all containers Nov 02 20:14:31.845097 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.845061 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/1.log" Nov 02 20:14:31.845464 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.845443 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/1.log" Nov 02 20:14:31.856720 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.856689 4971 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 20:14:31.858365 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.858340 4971 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 20:14:31.858365 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.858365 4971 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 20:14:31.858503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.858385 4971 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 20:14:31.858503 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:31.858418 4971 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 20:14:31.877730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.877707 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/1.log" Nov 02 20:14:31.877843 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.877740 4971 generic.go:356] "Generic (PLEG): container finished" podID="a6fea55b-74c2-40b1-9952-cd767e448317" containerID="05b60e41e161918b16b6c84ef6c992e2704d297642ef014aee8dfd11bbdfb89b" exitCode=255 Nov 02 20:14:31.882619 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.882589 4971 generic.go:356] "Generic (PLEG): container finished" podID="a138b9a8-1f14-42a3-97f6-9e2612f83470" containerID="67383d938a3ca80383ba6bc4a24da54c321cc3e4b4ede1ed449ec389de697968" exitCode=0 Nov 02 20:14:31.882619 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.882619 4971 generic.go:356] "Generic (PLEG): container finished" podID="a138b9a8-1f14-42a3-97f6-9e2612f83470" containerID="343e8c1f2d0e861a5f9a4a9deba0ac3befedf047a80020f8b3c75954636c5f03" exitCode=0 Nov 02 20:14:31.882750 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.882629 4971 generic.go:356] "Generic (PLEG): container finished" podID="a138b9a8-1f14-42a3-97f6-9e2612f83470" containerID="54295e0ef15084eaf2104c3c6e4bede2665a7d0d94e514eff60adac8c5007cbb" exitCode=0 Nov 02 20:14:31.882750 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.882640 4971 generic.go:356] "Generic (PLEG): container finished" podID="a138b9a8-1f14-42a3-97f6-9e2612f83470" containerID="e2685dcaa9535d0d0ba80e466996feffc2b99d56d55aa2571f31c80e934409ee" exitCode=0 Nov 02 20:14:31.882750 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.882649 4971 generic.go:356] "Generic (PLEG): container finished" podID="a138b9a8-1f14-42a3-97f6-9e2612f83470" containerID="dd9126013f33695c76a8fbb9a97d79bd2bb240618723e6c0a7ce4371cd63b1c9" exitCode=0 Nov 02 20:14:31.882750 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.882656 4971 generic.go:356] "Generic (PLEG): container finished" podID="a138b9a8-1f14-42a3-97f6-9e2612f83470" containerID="ab0e35440e23a1ac0c11ed1aca20824d5ad11e2d44ee51eaff92fd9a93c9a6df" exitCode=0 Nov 02 20:14:31.897774 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.897751 4971 generic.go:356] "Generic (PLEG): container finished" podID="4b86cdc381e2e2430d60e71836e38622" containerID="b2b37f6fb8b2176678cb69ccd70f080c5c5ad01d4eed8dc5c205924ce6adca37" exitCode=0 Nov 02 20:14:31.920427 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.920378 4971 manager.go:324] Recovery completed Nov 02 20:14:31.921689 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:31.921673 4971 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-120.us-west-2.compute.internal\" not found" Nov 02 20:14:31.937436 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.937420 4971 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:14:31.938578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.938563 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:14:31.938667 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.938592 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:14:31.938667 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.938602 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:14:31.941816 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.941792 4971 cpu_manager.go:226] "Starting" policy="none" Nov 02 20:14:31.941816 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.941815 4971 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 20:14:31.941941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.941839 4971 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 20:14:31.942057 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.942021 4971 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 20:14:31.942131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.942060 4971 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 20:14:31.942131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.942109 4971 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 20:14:31.942131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.942117 4971 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 20:14:31.942131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.942133 4971 policy_none.go:50] "Start" Nov 02 20:14:31.942309 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.942153 4971 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 20:14:31.942309 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.942167 4971 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 20:14:31.942390 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.942380 4971 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 20:14:31.942431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.942390 4971 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 20:14:31.942431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.942399 4971 policy_none.go:44] "Start" Nov 02 20:14:31.942522 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.942499 4971 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:14:31.943646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.943628 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:14:31.943718 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.943660 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:14:31.943718 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.943674 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:14:31.947379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.947356 4971 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 20:14:31.950440 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.950424 4971 manager.go:341] "Starting Device Plugin manager" Nov 02 20:14:31.950510 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:31.950450 4971 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 20:14:31.950510 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.950459 4971 server.go:87] "Starting device plugin registration server" Nov 02 20:14:31.950764 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.950752 4971 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 20:14:31.950795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.950769 4971 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 20:14:31.950867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.950845 4971 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 20:14:31.950990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.950949 4971 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 20:14:31.951121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.951017 4971 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 20:14:31.951121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.951026 4971 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 20:14:31.951255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.951077 4971 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T20:14:31.950984913Z","UUID":"586c8415-8c24-46b7-8bfd-4e214d53394a","Handler":null,"Name":"","Endpoint":""} Nov 02 20:14:31.952491 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:31.952473 4971 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 20:14:31.952600 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:31.952505 4971 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-120.us-west-2.compute.internal\" not found" Nov 02 20:14:31.952600 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.952516 4971 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 20:14:31.952600 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.952536 4971 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 20:14:31.959217 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.958549 4971 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal"] Nov 02 20:14:31.959340 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.959327 4971 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:14:31.960202 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.960185 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:14:31.960280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.960208 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:14:31.960280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.960218 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:14:31.960618 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.960601 4971 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:14:31.960811 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.960797 4971 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:14:31.961649 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.961632 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:14:31.961737 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.961664 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:14:31.961737 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.961660 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:14:31.961737 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.961681 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:14:31.961892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.961743 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:14:31.961892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.961760 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:14:31.961993 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:31.961914 4971 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-120.us-west-2.compute.internal\" not found" node="ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:31.962184 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.962168 4971 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:14:31.962251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.962188 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" event={"ID":"b918fb29af61c47ab1271b4e381400d1","Type":"ContainerStarted","Data":"01d201fb399e13b10ad3798372fda7bceb4995ddc731826c065c48d531abe0b8"} Nov 02 20:14:31.962251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.962235 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" event={"ID":"b918fb29af61c47ab1271b4e381400d1","Type":"ContainerStarted","Data":"8ea136b3030a2d7db3d157c0d72d2877085de68f36077b113c39ef0a8d689e27"} Nov 02 20:14:31.962359 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.962253 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" event={"ID":"4b86cdc381e2e2430d60e71836e38622","Type":"ContainerStarted","Data":"8b3b0eecee7ca50a29def95a73a68308d72a6da1803ab884e50396673c21d10a"} Nov 02 20:14:31.962359 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.962272 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" event={"ID":"4b86cdc381e2e2430d60e71836e38622","Type":"ContainerDied","Data":"b2b37f6fb8b2176678cb69ccd70f080c5c5ad01d4eed8dc5c205924ce6adca37"} Nov 02 20:14:31.962359 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.962291 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" event={"ID":"4b86cdc381e2e2430d60e71836e38622","Type":"ContainerStarted","Data":"690a5c20ac6d5a4dc1bf8a9ccf0448d9b7d97f3c5d154b65e2b20a6534b579e6"} Nov 02 20:14:31.962892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.962871 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:14:31.962985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.962908 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:14:31.962985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:31.962920 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:14:31.963115 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:31.963043 4971 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-120.us-west-2.compute.internal\" not found" node="ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.017715 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.017694 4971 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:14:32.017826 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.017699 4971 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:14:32.017826 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.017804 4971 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 20:14:32.051688 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.051667 4971 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:14:32.052516 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.052493 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:14:32.052620 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.052523 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:14:32.052620 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.052539 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:14:32.052620 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.052582 4971 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.066224 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.066196 4971 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:14:32.066517 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.066499 4971 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:14:32.118904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.118868 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b86cdc381e2e2430d60e71836e38622-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"4b86cdc381e2e2430d60e71836e38622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.119016 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.118904 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b86cdc381e2e2430d60e71836e38622-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"4b86cdc381e2e2430d60e71836e38622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.119016 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.118947 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b918fb29af61c47ab1271b4e381400d1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"b918fb29af61c47ab1271b4e381400d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.121994 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.121973 4971 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.156123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.156100 4971 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.156190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.156156 4971 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.156546 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.156533 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeNotReady" Nov 02 20:14:32.156598 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.156559 4971 setters.go:546] "Node became not ready" node="ip-10-0-140-120.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T20:14:32Z","lastTransitionTime":"2026-11-02T20:14:32Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 02 20:14:32.177381 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.177333 4971 nodeinfomanager.go:471] existing CSINode "ip-10-0-140-120.us-west-2.compute.internal" is owned by different node (oldNodeID="1c2ef97d-ed8d-4989-a1b5-e98efc6a22c6", newNodeID=""), cleaning up... Nov 02 20:14:32.183710 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.183691 4971 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 20:14:32.183757 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:32.183729 4971 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.183757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.183741 4971 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.219199 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.219181 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b86cdc381e2e2430d60e71836e38622-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"4b86cdc381e2e2430d60e71836e38622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.219286 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.219204 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b86cdc381e2e2430d60e71836e38622-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"4b86cdc381e2e2430d60e71836e38622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.219286 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.219232 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b918fb29af61c47ab1271b4e381400d1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"b918fb29af61c47ab1271b4e381400d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.219286 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.219266 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b918fb29af61c47ab1271b4e381400d1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"b918fb29af61c47ab1271b4e381400d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.219286 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.219232 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b86cdc381e2e2430d60e71836e38622-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"4b86cdc381e2e2430d60e71836e38622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.219437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.219265 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b86cdc381e2e2430d60e71836e38622-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal\" (UID: \"4b86cdc381e2e2430d60e71836e38622\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.241043 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.241023 4971 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 20:14:32.241135 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:32.241056 4971 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.809931 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.809854 4971 apiserver.go:52] "Watching apiserver" Nov 02 20:14:32.919293 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.919261 4971 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.919462 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.919385 4971 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.954314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.954291 4971 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 20:14:32.954947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.954921 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-rphxh","openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf","kube-system/global-pull-secret-syncer-jjtbq","kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal","openshift-ingress-canary/ingress-canary-278nr","kube-system/kas-connection-checker-685895c7c9-5bk6w","openshift-insights/periodic-gathering-hxckx-j8tzv","openshift-image-registry/image-registry-5d646d5849-btrvm","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r","openshift-multus/multus-gxrvt","kube-system/kas-connection-checker-685895c7c9-7fd7l","openshift-cluster-node-tuning-operator/tuned-fv8rs","openshift-network-diagnostics/network-check-source-6f78d59f99-6tfzr","openshift-network-operator/iptables-alerter-l54nk","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6","kube-system/konnectivity-agent-jbk5q","openshift-console-operator/console-operator-7cd56b8dbc-2kxv2","openshift-ingress/router-default-f6f5bd44c-4tksg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm","openshift-multus/multus-additional-cni-plugins-bvfmm","openshift-insights/insights-operator-7c4986d449-v9wrk","openshift-ovn-kubernetes/ovnkube-node-9ngnp","openshift-multus/network-metrics-daemon-5kzc7","openshift-service-ca/service-ca-5fffc97bdf-4n9mb","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-bgd5x","openshift-dns/node-resolver-26nmv","openshift-network-console/networking-console-plugin-86c478bb88-89j9l","openshift-insights/insights-runtime-extractor-l4q8k","openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm","kube-system/kas-connection-checker-685895c7c9-7shr8","openshift-image-registry/node-ca-7pl28","openshift-dns/dns-default-8lr5b","openshift-image-registry/image-registry-6578786844-fgcrv"] Nov 02 20:14:32.957587 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.957566 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:32.957985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.957961 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:32.958588 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.958569 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:32.958985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.958970 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:32.959323 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.959301 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:32.983191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.965367 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:32.983191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.970978 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:32.983191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.971889 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:32.983191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.974020 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:32.983191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.977679 4971 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 20:14:32.983191 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:32.977737 4971 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:32.983191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:32.978383 4971 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 20:14:32.983191 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:32.978435 4971 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.us-west-2.compute.internal" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.011653 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.011727 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.011791 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.011797 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.011898 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012109 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012161 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012194 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012323 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012355 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012373 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012388 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012481 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012488 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012543 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012547 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012568 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012575 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012598 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012612 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012627 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012630 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012542 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012644 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012614 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012669 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-q2gdn\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012482 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.012756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.012777 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.014127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.014105 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.014281 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.014254 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gv6xl\"" Nov 02 20:14:33.015149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.014339 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.015149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.014701 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ng8kx\"" Nov 02 20:14:33.015149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.014785 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.015149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.015015 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.015149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.015071 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-r6qc8\"" Nov 02 20:14:33.015149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.015149 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hfjnv\"" Nov 02 20:14:33.015428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.015203 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 20:14:33.015428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.015253 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.015428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.015310 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fjr76\"" Nov 02 20:14:33.024627 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.024603 4971 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 20:14:33.025015 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.024997 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-trusted-ca\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.025107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025032 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-ovn\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.025107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025055 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-log-socket\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.025107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025096 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.025257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025211 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg6gq\" (UniqueName: \"kubernetes.io/projected/22cd77ec-a3f8-4e80-8ffb-def069d70e34-kube-api-access-qg6gq\") pod \"kas-connection-checker-685895c7c9-7fd7l\" (UID: \"22cd77ec-a3f8-4e80-8ffb-def069d70e34\") " pod="kube-system/kas-connection-checker-685895c7c9-7fd7l" Nov 02 20:14:33.025306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025253 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.025306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025286 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-registration-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.025402 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025311 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-sys\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.025402 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025336 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-system-cni-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.025402 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025361 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-cnibin\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.025542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025411 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:33.025542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025438 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-sys-fs\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.025542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025473 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngldn\" (UniqueName: \"kubernetes.io/projected/283c8ed3-38ac-4a45-82d0-ec5bd9245376-kube-api-access-ngldn\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:33.025542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025497 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snn5k\" (UniqueName: \"kubernetes.io/projected/bcc543bf-8ffd-488d-999a-2b8da64e9c31-kube-api-access-snn5k\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.025542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025529 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f65dc226-70a9-4fcf-ae3b-28559c2ff235-service-ca-bundle\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:33.025778 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025554 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-installation-pull-secrets\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.025778 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025599 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mk86\" (UniqueName: \"kubernetes.io/projected/896fb858-68d1-462c-8a00-2b2b6b5a01af-kube-api-access-6mk86\") pod \"service-ca-operator-544d749c8c-6qvvf\" (UID: \"896fb858-68d1-462c-8a00-2b2b6b5a01af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:33.025778 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025635 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll47d\" (UniqueName: \"kubernetes.io/projected/6fea26b0-2cef-4683-a20f-620ef0880bfc-kube-api-access-ll47d\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:33.025778 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025743 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysctl-conf\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.025964 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025785 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f65dc226-70a9-4fcf-ae3b-28559c2ff235-service-ca-bundle\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:33.025964 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025905 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tkr\" (UniqueName: \"kubernetes.io/projected/bfe84946-82e9-456a-b81f-af887093749b-kube-api-access-b6tkr\") pod \"network-check-source-6f78d59f99-6tfzr\" (UID: \"bfe84946-82e9-456a-b81f-af887093749b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tfzr" Nov 02 20:14:33.025964 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.025951 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-cni-binary-copy\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.026132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026051 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99e51c7f-420e-4709-b098-2ac9f0eaaedb-cert\") pod \"ingress-canary-278nr\" (UID: \"99e51c7f-420e-4709-b098-2ac9f0eaaedb\") " pod="openshift-ingress-canary/ingress-canary-278nr" Nov 02 20:14:33.026132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026100 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:33.026229 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026132 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-cni-multus\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.026229 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026160 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nczq\" (UniqueName: \"kubernetes.io/projected/eb120384-9c46-4dbd-bb80-bedf4bbe957c-kube-api-access-5nczq\") pod \"service-ca-5fffc97bdf-4n9mb\" (UID: \"eb120384-9c46-4dbd-bb80-bedf4bbe957c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:33.026229 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026183 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-modprobe-d\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.026372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026230 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-run\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.026372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026354 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-trusted-ca-bundle\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:33.026465 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026431 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltqcp\" (UniqueName: \"kubernetes.io/projected/4d75d87e-395e-4668-8b7a-42bdd79d08d9-kube-api-access-ltqcp\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:33.026516 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026461 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwzl\" (UniqueName: \"kubernetes.io/projected/aef015e0-b0a9-4c9f-8140-dd74d8265e64-kube-api-access-tfwzl\") pod \"kas-connection-checker-685895c7c9-5bk6w\" (UID: \"aef015e0-b0a9-4c9f-8140-dd74d8265e64\") " pod="kube-system/kas-connection-checker-685895c7c9-5bk6w" Nov 02 20:14:33.026516 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026487 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-node-log\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.026619 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026512 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:33.026619 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026539 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.026619 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026567 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.026619 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026589 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-socket-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.026619 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026615 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-service-ca-bundle\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:33.026852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026648 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-bound-sa-token\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.026852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026673 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-kubelet\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.026852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026703 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-lib-modules\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.026852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026728 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-default-certificate\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:33.026852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026754 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-etc-kubernetes\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.026852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026794 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f65dc226-70a9-4fcf-ae3b-28559c2ff235-serving-cert\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:33.026852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026831 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2xgj\" (UniqueName: \"kubernetes.io/projected/a6fea55b-74c2-40b1-9952-cd767e448317-kube-api-access-f2xgj\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:33.027190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026864 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c868547-5957-44d5-99d3-4aefc664abb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:33.027190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026889 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-netns\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.027190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026916 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-certificates\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.027190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026940 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-ovnkube-script-lib\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.027190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026959 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f65dc226-70a9-4fcf-ae3b-28559c2ff235-serving-cert\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:33.027190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026963 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-serviceca\") pod \"node-ca-7pl28\" (UID: \"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd\") " pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:14:33.027190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.026995 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a6fea55b-74c2-40b1-9952-cd767e448317-config\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:33.027190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027028 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6fea26b0-2cef-4683-a20f-620ef0880bfc-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:33.027190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027057 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68a10036-5c78-4b01-b06d-075a727a5d0b-host-slash\") pod \"iptables-alerter-l54nk\" (UID: \"68a10036-5c78-4b01-b06d-075a727a5d0b\") " pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:14:33.027190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027118 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgpnn\" (UniqueName: \"kubernetes.io/projected/30d6b618-cc4e-48b8-a7b4-9649a4242aad-kube-api-access-bgpnn\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:33.027190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027160 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/eb120384-9c46-4dbd-bb80-bedf4bbe957c-signing-cabundle\") pod \"service-ca-5fffc97bdf-4n9mb\" (UID: \"eb120384-9c46-4dbd-bb80-bedf4bbe957c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:33.027190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027186 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027204 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a6fea55b-74c2-40b1-9952-cd767e448317-config\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027209 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-host\") pod \"node-ca-7pl28\" (UID: \"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd\") " pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027234 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-stats-auth\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027270 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5mrg\" (UniqueName: \"kubernetes.io/projected/a138b9a8-1f14-42a3-97f6-9e2612f83470-kube-api-access-x5mrg\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027296 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4d75d87e-395e-4668-8b7a-42bdd79d08d9-archives-path\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027322 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf87f\" (UniqueName: \"kubernetes.io/projected/39e68d28-771c-48e8-ba38-f8aad80a6bf9-kube-api-access-jf87f\") pod \"node-resolver-26nmv\" (UID: \"39e68d28-771c-48e8-ba38-f8aad80a6bf9\") " pod="openshift-dns/node-resolver-26nmv" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027347 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-device-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027372 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027395 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/283c8ed3-38ac-4a45-82d0-ec5bd9245376-config-volume\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027420 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpz59\" (UniqueName: \"kubernetes.io/projected/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-kube-api-access-gpz59\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027461 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/39e68d28-771c-48e8-ba38-f8aad80a6bf9-hosts-file\") pod \"node-resolver-26nmv\" (UID: \"39e68d28-771c-48e8-ba38-f8aad80a6bf9\") " pod="openshift-dns/node-resolver-26nmv" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027486 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e68d28-771c-48e8-ba38-f8aad80a6bf9-tmp-dir\") pod \"node-resolver-26nmv\" (UID: \"39e68d28-771c-48e8-ba38-f8aad80a6bf9\") " pod="openshift-dns/node-resolver-26nmv" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027510 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b51723af-6a33-4535-91c0-905ce2a51b24-agent-certs\") pod \"konnectivity-agent-jbk5q\" (UID: \"b51723af-6a33-4535-91c0-905ce2a51b24\") " pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027534 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw5x4\" (UniqueName: \"kubernetes.io/projected/64f527e7-dcc8-49be-8c15-a078a7aed186-kube-api-access-gw5x4\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027543 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-stats-auth\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027558 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-daemon-config\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027603 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f65dc226-70a9-4fcf-ae3b-28559c2ff235-snapshots\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027634 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-installation-pull-secrets\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027664 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64f527e7-dcc8-49be-8c15-a078a7aed186-ovn-node-metrics-cert\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027691 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysconfig\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027730 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-var-lib-kubelet\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027758 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-cnibin\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027785 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f65dc226-70a9-4fcf-ae3b-28559c2ff235-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:33.027824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027823 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02eded99-edf7-41a0-8866-549252be5fe0-ca-trust-extracted\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.028944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027847 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-var-lib-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.028944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027876 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-ovnkube-config\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.028944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027921 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-cni-bin\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.028944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027952 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzbsq\" (UniqueName: \"kubernetes.io/projected/5013aa7f-b2fd-4df0-b5e3-3478bef510ee-kube-api-access-kzbsq\") pod \"kas-connection-checker-685895c7c9-7shr8\" (UID: \"5013aa7f-b2fd-4df0-b5e3-3478bef510ee\") " pod="kube-system/kas-connection-checker-685895c7c9-7shr8" Nov 02 20:14:33.028944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.027979 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-run-ovn-kubernetes\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.028944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.028005 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/eb120384-9c46-4dbd-bb80-bedf4bbe957c-signing-key\") pod \"service-ca-5fffc97bdf-4n9mb\" (UID: \"eb120384-9c46-4dbd-bb80-bedf4bbe957c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:33.028944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.028031 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysctl-d\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.028944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.028056 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcc543bf-8ffd-488d-999a-2b8da64e9c31-tmp\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.028944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.028107 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:33.028944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.028144 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fec747e-0efa-40f5-afaf-d45d820700b1-cni-binary-copy\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.028944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.028160 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4d75d87e-395e-4668-8b7a-42bdd79d08d9-archives-path\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:33.028944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.028172 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fknt\" (UniqueName: \"kubernetes.io/projected/f22a4aac-bd75-41d2-b1c2-9293bb405389-kube-api-access-2fknt\") pod \"volume-data-source-validator-58c9f5f46d-bgd5x\" (UID: \"f22a4aac-bd75-41d2-b1c2-9293bb405389\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-bgd5x" Nov 02 20:14:33.028944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.028202 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d23cf40e-53e9-406e-92f8-ce628b1baa68-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6\" (UID: \"d23cf40e-53e9-406e-92f8-ce628b1baa68\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:33.028944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.028230 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b51723af-6a33-4535-91c0-905ce2a51b24-konnectivity-ca\") pod \"konnectivity-agent-jbk5q\" (UID: \"b51723af-6a33-4535-91c0-905ce2a51b24\") " pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:14:33.028944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.028622 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-default-certificate\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:33.028944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.028912 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f65dc226-70a9-4fcf-ae3b-28559c2ff235-snapshots\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:33.029730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.029025 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-bound-sa-token\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.029730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.029293 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02eded99-edf7-41a0-8866-549252be5fe0-ca-trust-extracted\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.029730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.029307 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:33.029730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.029342 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcc543bf-8ffd-488d-999a-2b8da64e9c31-tmp\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.029730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.029598 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-systemd-units\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.029730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.029623 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d23cf40e-53e9-406e-92f8-ce628b1baa68-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6\" (UID: \"d23cf40e-53e9-406e-92f8-ce628b1baa68\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:33.029933 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.029773 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7krck\" (UniqueName: \"kubernetes.io/projected/2c868547-5957-44d5-99d3-4aefc664abb1-kube-api-access-7krck\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:33.029933 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.029803 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/afe07c4f-7282-4239-9761-c61ef4153cd9-kubelet-config\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:33.029933 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.029850 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-slash\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.029933 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.029878 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-systemd\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.029933 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.029904 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/283c8ed3-38ac-4a45-82d0-ec5bd9245376-tmp-dir\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:33.029933 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.029929 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-cni-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.030139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.029986 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cb81e57-c7dc-4447-959f-d5496ffee976-ca-trust-extracted\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.030139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030012 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-etc-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.030139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030034 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-cni-netd\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.030139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030056 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c868547-5957-44d5-99d3-4aefc664abb1-crio-socket\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:33.030139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030099 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c868547-5957-44d5-99d3-4aefc664abb1-data-volume\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:33.030139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030131 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-env-overrides\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.030297 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030166 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-socket-dir-parent\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.030297 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030191 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-kubelet\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.030297 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030267 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/afe07c4f-7282-4239-9761-c61ef4153cd9-dbus\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:33.030380 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030303 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9d5e73f9-907c-40b1-a596-b079ded3abfb-nginx-conf\") pod \"networking-console-plugin-86c478bb88-89j9l\" (UID: \"9d5e73f9-907c-40b1-a596-b079ded3abfb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:33.030380 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030329 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.030380 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030353 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-k8s-cni-cncf-io\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.030486 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030379 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-cni-bin\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.030486 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030410 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-89j9l\" (UID: \"9d5e73f9-907c-40b1-a596-b079ded3abfb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:33.030486 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030436 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-system-cni-dir\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.030486 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030465 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8r65\" (UniqueName: \"kubernetes.io/projected/d23cf40e-53e9-406e-92f8-ce628b1baa68-kube-api-access-d8r65\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6\" (UID: \"d23cf40e-53e9-406e-92f8-ce628b1baa68\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:33.030597 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030493 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t22cd\" (UniqueName: \"kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd\") pod \"network-check-target-rphxh\" (UID: \"f8dd1189-9c04-4ba1-a349-04e7dabcab52\") " pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:33.030597 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030517 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-trusted-ca\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.030597 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030541 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-kubernetes\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.030597 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030566 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:33.030597 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030592 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx5xx\" (UniqueName: \"kubernetes.io/projected/99e51c7f-420e-4709-b098-2ac9f0eaaedb-kube-api-access-cx5xx\") pod \"ingress-canary-278nr\" (UID: \"99e51c7f-420e-4709-b098-2ac9f0eaaedb\") " pod="openshift-ingress-canary/ingress-canary-278nr" Nov 02 20:14:33.030725 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030618 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7nbm\" (UniqueName: \"kubernetes.io/projected/68a10036-5c78-4b01-b06d-075a727a5d0b-kube-api-access-s7nbm\") pod \"iptables-alerter-l54nk\" (UID: \"68a10036-5c78-4b01-b06d-075a727a5d0b\") " pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:14:33.030725 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030647 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-image-registry-private-configuration\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.030725 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030673 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpqp8\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-kube-api-access-xpqp8\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.030812 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030803 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-host\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.030845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030828 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-conf-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.030874 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030855 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcq6w\" (UniqueName: \"kubernetes.io/projected/9fec747e-0efa-40f5-afaf-d45d820700b1-kube-api-access-wcq6w\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.030907 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030882 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f65dc226-70a9-4fcf-ae3b-28559c2ff235-tmp\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:33.030937 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030909 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bskm\" (UniqueName: \"kubernetes.io/projected/f65dc226-70a9-4fcf-ae3b-28559c2ff235-kube-api-access-8bskm\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:33.030966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030936 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmzm\" (UniqueName: \"kubernetes.io/projected/1e96f673-1cf6-48f1-a55d-347662c75b16-kube-api-access-qvmzm\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.030996 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030963 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-registry-certificates\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.030996 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.030990 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmsvc\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-kube-api-access-kmsvc\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.031052 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031015 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d23cf40e-53e9-406e-92f8-ce628b1baa68-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6\" (UID: \"d23cf40e-53e9-406e-92f8-ce628b1baa68\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:33.031052 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031045 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68a10036-5c78-4b01-b06d-075a727a5d0b-iptables-alerter-script\") pod \"iptables-alerter-l54nk\" (UID: \"68a10036-5c78-4b01-b06d-075a727a5d0b\") " pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:14:33.031129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031071 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:33.031160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031126 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.031160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031153 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-image-registry-private-configuration\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.031218 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031180 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:33.031218 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031204 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-tuned\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.031272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031230 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8mlg\" (UniqueName: \"kubernetes.io/projected/cb6295a6-f31e-48a3-b4df-7d872a961161-kube-api-access-v8mlg\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:33.031272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031257 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-etc-selinux\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.031329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031281 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-hostroot\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.031329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031306 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.031386 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031331 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-run-netns\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.031386 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031356 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-systemd\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.031443 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031383 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd99v\" (UniqueName: \"kubernetes.io/projected/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-kube-api-access-hd99v\") pod \"node-ca-7pl28\" (UID: \"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd\") " pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:14:33.031443 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031411 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:33.031443 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031437 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/896fb858-68d1-462c-8a00-2b2b6b5a01af-config\") pod \"service-ca-operator-544d749c8c-6qvvf\" (UID: \"896fb858-68d1-462c-8a00-2b2b6b5a01af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:33.031564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031464 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6fea55b-74c2-40b1-9952-cd767e448317-trusted-ca\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:33.031564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031492 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-multus-certs\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.031564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031519 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/896fb858-68d1-462c-8a00-2b2b6b5a01af-serving-cert\") pod \"service-ca-operator-544d749c8c-6qvvf\" (UID: \"896fb858-68d1-462c-8a00-2b2b6b5a01af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:33.031564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031545 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a6fea55b-74c2-40b1-9952-cd767e448317-serving-cert\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:33.031734 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.031718 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a6fea55b-74c2-40b1-9952-cd767e448317-serving-cert\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:33.032436 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.032418 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/283c8ed3-38ac-4a45-82d0-ec5bd9245376-tmp-dir\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:33.032497 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.032483 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cb81e57-c7dc-4447-959f-d5496ffee976-ca-trust-extracted\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.032558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.032546 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c868547-5957-44d5-99d3-4aefc664abb1-data-volume\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:33.033539 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.033520 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f65dc226-70a9-4fcf-ae3b-28559c2ff235-tmp\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:33.034140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.034116 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d23cf40e-53e9-406e-92f8-ce628b1baa68-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6\" (UID: \"d23cf40e-53e9-406e-92f8-ce628b1baa68\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:33.034278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.034262 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-tuned\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.034632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.034610 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/896fb858-68d1-462c-8a00-2b2b6b5a01af-config\") pod \"service-ca-operator-544d749c8c-6qvvf\" (UID: \"896fb858-68d1-462c-8a00-2b2b6b5a01af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:33.034813 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.034794 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/896fb858-68d1-462c-8a00-2b2b6b5a01af-serving-cert\") pod \"service-ca-operator-544d749c8c-6qvvf\" (UID: \"896fb858-68d1-462c-8a00-2b2b6b5a01af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:33.035053 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.035035 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-service-ca-bundle\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:33.035130 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.035115 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e68d28-771c-48e8-ba38-f8aad80a6bf9-tmp-dir\") pod \"node-resolver-26nmv\" (UID: \"39e68d28-771c-48e8-ba38-f8aad80a6bf9\") " pod="openshift-dns/node-resolver-26nmv" Nov 02 20:14:33.059331 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.059309 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 20:14:33.059544 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.059527 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.059601 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.059583 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 20:14:33.059664 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.059646 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.059711 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.059694 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 20:14:33.059756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.059729 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.059756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.059739 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 20:14:33.059846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.059801 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.060180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.059954 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 20:14:33.060180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.059956 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.060180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.059973 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 20:14:33.060180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060038 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.060180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060050 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.060180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060062 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5v4kq\"" Nov 02 20:14:33.060180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060110 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.060180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060148 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.060180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060163 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.060603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060214 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 20:14:33.060603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060223 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 20:14:33.060603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060283 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 20:14:33.060603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060291 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 20:14:33.060603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060348 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.060603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060363 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.060603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060460 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 20:14:33.060603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060503 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 20:14:33.060603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060573 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.060984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060719 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.060984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.060951 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 20:14:33.061142 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061101 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 20:14:33.061142 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061118 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4wc8w\"" Nov 02 20:14:33.061234 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061142 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 20:14:33.061234 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061153 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 20:14:33.061234 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061166 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 20:14:33.061234 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061201 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 20:14:33.061234 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061214 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ldshw\"" Nov 02 20:14:33.061449 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061257 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7bq62\"" Nov 02 20:14:33.061449 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061276 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-t6dvk\"" Nov 02 20:14:33.061449 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061301 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5vs7j\"" Nov 02 20:14:33.061576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061523 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 20:14:33.061576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061571 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 20:14:33.061666 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061628 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-xtqwc\"" Nov 02 20:14:33.061737 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061691 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.061783 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061746 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.061840 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061822 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 20:14:33.061910 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.061897 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.062024 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062012 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d6nbh\"" Nov 02 20:14:33.062111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062076 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vjrtf\"" Nov 02 20:14:33.062179 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062126 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xxqx9\"" Nov 02 20:14:33.062226 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062178 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7m5tr\"" Nov 02 20:14:33.062226 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062194 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.062314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062228 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 20:14:33.062314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062274 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 20:14:33.062314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062282 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hxslf\"" Nov 02 20:14:33.062314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062285 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-74k7h\"" Nov 02 20:14:33.062495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062329 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 20:14:33.062495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062376 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 20:14:33.062495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062389 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 20:14:33.062495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062413 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 20:14:33.062495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062331 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s29pz\"" Nov 02 20:14:33.062495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062438 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-djbmg\"" Nov 02 20:14:33.062745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062509 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 20:14:33.062745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062517 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9bjnj\"" Nov 02 20:14:33.062745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062556 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 20:14:33.062745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062563 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 20:14:33.062745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062578 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qfmz2\"" Nov 02 20:14:33.062745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062598 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 20:14:33.062745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062657 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zgrcn\"" Nov 02 20:14:33.062745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062713 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 20:14:33.063129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062808 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 20:14:33.063129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062834 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kz985\"" Nov 02 20:14:33.063129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.062873 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-env-overrides\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.063129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.063003 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-82tkg\"" Nov 02 20:14:33.063129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.063032 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9d5e73f9-907c-40b1-a596-b079ded3abfb-nginx-conf\") pod \"networking-console-plugin-86c478bb88-89j9l\" (UID: \"9d5e73f9-907c-40b1-a596-b079ded3abfb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:33.063618 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.063583 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-trusted-ca\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.063697 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.063679 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-image-registry-private-configuration\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.064874 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.064854 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-registry-certificates\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.065044 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.065023 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68a10036-5c78-4b01-b06d-075a727a5d0b-iptables-alerter-script\") pod \"iptables-alerter-l54nk\" (UID: \"68a10036-5c78-4b01-b06d-075a727a5d0b\") " pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:14:33.065186 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.065168 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b51723af-6a33-4535-91c0-905ce2a51b24-agent-certs\") pod \"konnectivity-agent-jbk5q\" (UID: \"b51723af-6a33-4535-91c0-905ce2a51b24\") " pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:14:33.065275 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.065260 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/afe07c4f-7282-4239-9761-c61ef4153cd9-original-pull-secret\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:33.065436 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.065418 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6fea55b-74c2-40b1-9952-cd767e448317-trusted-ca\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:33.065525 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.065507 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-trusted-ca\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.065650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.065631 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.065795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.065779 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-installation-pull-secrets\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.065864 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.065847 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-image-registry-private-configuration\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.066803 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.066779 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99e51c7f-420e-4709-b098-2ac9f0eaaedb-cert\") pod \"ingress-canary-278nr\" (UID: \"99e51c7f-420e-4709-b098-2ac9f0eaaedb\") " pod="openshift-ingress-canary/ingress-canary-278nr" Nov 02 20:14:33.067150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.067129 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-trusted-ca-bundle\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:33.071123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.070313 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-serviceca\") pod \"node-ca-7pl28\" (UID: \"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd\") " pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:14:33.071123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.070330 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.071123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.070378 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a138b9a8-1f14-42a3-97f6-9e2612f83470-cni-binary-copy\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.071123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.070458 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-certificates\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.071123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.070573 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-daemon-config\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.071123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.070628 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b51723af-6a33-4535-91c0-905ce2a51b24-konnectivity-ca\") pod \"konnectivity-agent-jbk5q\" (UID: \"b51723af-6a33-4535-91c0-905ce2a51b24\") " pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:14:33.071123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.070686 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-ovnkube-script-lib\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.071123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.070747 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/283c8ed3-38ac-4a45-82d0-ec5bd9245376-config-volume\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:33.071123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.070778 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-installation-pull-secrets\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.071123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.070869 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64f527e7-dcc8-49be-8c15-a078a7aed186-ovnkube-config\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.071123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.070900 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64f527e7-dcc8-49be-8c15-a078a7aed186-ovn-node-metrics-cert\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.071123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.070952 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/eb120384-9c46-4dbd-bb80-bedf4bbe957c-signing-key\") pod \"service-ca-5fffc97bdf-4n9mb\" (UID: \"eb120384-9c46-4dbd-bb80-bedf4bbe957c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:33.071686 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.071140 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fec747e-0efa-40f5-afaf-d45d820700b1-cni-binary-copy\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.071686 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.071302 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/eb120384-9c46-4dbd-bb80-bedf4bbe957c-signing-cabundle\") pod \"service-ca-5fffc97bdf-4n9mb\" (UID: \"eb120384-9c46-4dbd-bb80-bedf4bbe957c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:33.071686 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.071636 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6fea26b0-2cef-4683-a20f-620ef0880bfc-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:33.072852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.072262 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f65dc226-70a9-4fcf-ae3b-28559c2ff235-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:33.080992 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.080970 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg6gq\" (UniqueName: \"kubernetes.io/projected/22cd77ec-a3f8-4e80-8ffb-def069d70e34-kube-api-access-qg6gq\") pod \"kas-connection-checker-685895c7c9-7fd7l\" (UID: \"22cd77ec-a3f8-4e80-8ffb-def069d70e34\") " pod="kube-system/kas-connection-checker-685895c7c9-7fd7l" Nov 02 20:14:33.081331 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.081296 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngldn\" (UniqueName: \"kubernetes.io/projected/283c8ed3-38ac-4a45-82d0-ec5bd9245376-kube-api-access-ngldn\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:33.081399 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.081327 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snn5k\" (UniqueName: \"kubernetes.io/projected/bcc543bf-8ffd-488d-999a-2b8da64e9c31-kube-api-access-snn5k\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.082545 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.082519 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mk86\" (UniqueName: \"kubernetes.io/projected/896fb858-68d1-462c-8a00-2b2b6b5a01af-kube-api-access-6mk86\") pod \"service-ca-operator-544d749c8c-6qvvf\" (UID: \"896fb858-68d1-462c-8a00-2b2b6b5a01af\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" Nov 02 20:14:33.082971 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.082955 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltqcp\" (UniqueName: \"kubernetes.io/projected/4d75d87e-395e-4668-8b7a-42bdd79d08d9-kube-api-access-ltqcp\") pod \"periodic-gathering-hxckx-j8tzv\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:14:33.083439 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.083417 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwzl\" (UniqueName: \"kubernetes.io/projected/aef015e0-b0a9-4c9f-8140-dd74d8265e64-kube-api-access-tfwzl\") pod \"kas-connection-checker-685895c7c9-5bk6w\" (UID: \"aef015e0-b0a9-4c9f-8140-dd74d8265e64\") " pod="kube-system/kas-connection-checker-685895c7c9-5bk6w" Nov 02 20:14:33.083806 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.083785 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll47d\" (UniqueName: \"kubernetes.io/projected/6fea26b0-2cef-4683-a20f-620ef0880bfc-kube-api-access-ll47d\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:33.083858 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.083789 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tkr\" (UniqueName: \"kubernetes.io/projected/bfe84946-82e9-456a-b81f-af887093749b-kube-api-access-b6tkr\") pod \"network-check-source-6f78d59f99-6tfzr\" (UID: \"bfe84946-82e9-456a-b81f-af887093749b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tfzr" Nov 02 20:14:33.084065 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.084047 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nczq\" (UniqueName: \"kubernetes.io/projected/eb120384-9c46-4dbd-bb80-bedf4bbe957c-kube-api-access-5nczq\") pod \"service-ca-5fffc97bdf-4n9mb\" (UID: \"eb120384-9c46-4dbd-bb80-bedf4bbe957c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4n9mb" Nov 02 20:14:33.084367 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.084352 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2xgj\" (UniqueName: \"kubernetes.io/projected/a6fea55b-74c2-40b1-9952-cd767e448317-kube-api-access-f2xgj\") pod \"console-operator-7cd56b8dbc-2kxv2\" (UID: \"a6fea55b-74c2-40b1-9952-cd767e448317\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:33.086960 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.086939 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5mrg\" (UniqueName: \"kubernetes.io/projected/a138b9a8-1f14-42a3-97f6-9e2612f83470-kube-api-access-x5mrg\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.090742 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.090719 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-bound-sa-token\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.090812 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.090740 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-bound-sa-token\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.091494 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.091468 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzbsq\" (UniqueName: \"kubernetes.io/projected/5013aa7f-b2fd-4df0-b5e3-3478bef510ee-kube-api-access-kzbsq\") pod \"kas-connection-checker-685895c7c9-7shr8\" (UID: \"5013aa7f-b2fd-4df0-b5e3-3478bef510ee\") " pod="kube-system/kas-connection-checker-685895c7c9-7shr8" Nov 02 20:14:33.091704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.091686 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgpnn\" (UniqueName: \"kubernetes.io/projected/30d6b618-cc4e-48b8-a7b4-9649a4242aad-kube-api-access-bgpnn\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:33.091813 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.091763 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpz59\" (UniqueName: \"kubernetes.io/projected/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-kube-api-access-gpz59\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:33.091813 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.091766 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fknt\" (UniqueName: \"kubernetes.io/projected/f22a4aac-bd75-41d2-b1c2-9293bb405389-kube-api-access-2fknt\") pod \"volume-data-source-validator-58c9f5f46d-bgd5x\" (UID: \"f22a4aac-bd75-41d2-b1c2-9293bb405389\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-bgd5x" Nov 02 20:14:33.092155 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.092135 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf87f\" (UniqueName: \"kubernetes.io/projected/39e68d28-771c-48e8-ba38-f8aad80a6bf9-kube-api-access-jf87f\") pod \"node-resolver-26nmv\" (UID: \"39e68d28-771c-48e8-ba38-f8aad80a6bf9\") " pod="openshift-dns/node-resolver-26nmv" Nov 02 20:14:33.092282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.092263 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpqp8\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-kube-api-access-xpqp8\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.092831 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.092810 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmzm\" (UniqueName: \"kubernetes.io/projected/1e96f673-1cf6-48f1-a55d-347662c75b16-kube-api-access-qvmzm\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.095790 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.095770 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7krck\" (UniqueName: \"kubernetes.io/projected/2c868547-5957-44d5-99d3-4aefc664abb1-kube-api-access-7krck\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:33.097644 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.097619 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd99v\" (UniqueName: \"kubernetes.io/projected/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-kube-api-access-hd99v\") pod \"node-ca-7pl28\" (UID: \"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd\") " pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:14:33.097644 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.097633 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t22cd\" (UniqueName: \"kubernetes.io/projected/f8dd1189-9c04-4ba1-a349-04e7dabcab52-kube-api-access-t22cd\") pod \"network-check-target-rphxh\" (UID: \"f8dd1189-9c04-4ba1-a349-04e7dabcab52\") " pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:33.098189 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.098164 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bskm\" (UniqueName: \"kubernetes.io/projected/f65dc226-70a9-4fcf-ae3b-28559c2ff235-kube-api-access-8bskm\") pod \"insights-operator-7c4986d449-v9wrk\" (UID: \"f65dc226-70a9-4fcf-ae3b-28559c2ff235\") " pod="openshift-insights/insights-operator-7c4986d449-v9wrk" Nov 02 20:14:33.098272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.098209 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw5x4\" (UniqueName: \"kubernetes.io/projected/64f527e7-dcc8-49be-8c15-a078a7aed186-kube-api-access-gw5x4\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.098463 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.098440 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx5xx\" (UniqueName: \"kubernetes.io/projected/99e51c7f-420e-4709-b098-2ac9f0eaaedb-kube-api-access-cx5xx\") pod \"ingress-canary-278nr\" (UID: \"99e51c7f-420e-4709-b098-2ac9f0eaaedb\") " pod="openshift-ingress-canary/ingress-canary-278nr" Nov 02 20:14:33.098676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.098653 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcq6w\" (UniqueName: \"kubernetes.io/projected/9fec747e-0efa-40f5-afaf-d45d820700b1-kube-api-access-wcq6w\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.098815 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.098798 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8mlg\" (UniqueName: \"kubernetes.io/projected/cb6295a6-f31e-48a3-b4df-7d872a961161-kube-api-access-v8mlg\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:33.098917 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.098854 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmsvc\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-kube-api-access-kmsvc\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.099012 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.098894 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8r65\" (UniqueName: \"kubernetes.io/projected/d23cf40e-53e9-406e-92f8-ce628b1baa68-kube-api-access-d8r65\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6\" (UID: \"d23cf40e-53e9-406e-92f8-ce628b1baa68\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" Nov 02 20:14:33.099136 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.099063 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7nbm\" (UniqueName: \"kubernetes.io/projected/68a10036-5c78-4b01-b06d-075a727a5d0b-kube-api-access-s7nbm\") pod \"iptables-alerter-l54nk\" (UID: \"68a10036-5c78-4b01-b06d-075a727a5d0b\") " pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:14:33.132892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.132867 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysconfig\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.133030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.132903 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-var-lib-kubelet\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.133030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.132925 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-cnibin\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.133030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.132956 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-var-lib-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.133030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.132980 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-cni-bin\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.133030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133002 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysconfig\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.133030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133002 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-run-ovn-kubernetes\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133040 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysctl-d\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133067 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-cnibin\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133066 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-systemd-units\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133122 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/afe07c4f-7282-4239-9761-c61ef4153cd9-kubelet-config\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133126 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-cni-bin\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133143 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-slash\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133163 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-systemd\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133170 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-var-lib-kubelet\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133185 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-cni-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133211 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-etc-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133233 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-cni-netd\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133242 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-systemd-units\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133278 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-cni-netd\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133281 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-run-ovn-kubernetes\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133282 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-slash\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133300 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c868547-5957-44d5-99d3-4aefc664abb1-crio-socket\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133334 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-socket-dir-parent\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133336 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c868547-5957-44d5-99d3-4aefc664abb1-crio-socket\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133359 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-kubelet\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133382 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/afe07c4f-7282-4239-9761-c61ef4153cd9-dbus\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133387 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-socket-dir-parent\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133391 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysctl-d\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.133417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133409 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133431 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-systemd\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133430 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/afe07c4f-7282-4239-9761-c61ef4153cd9-kubelet-config\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133433 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-k8s-cni-cncf-io\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133461 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-etc-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133462 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-k8s-cni-cncf-io\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133469 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-cni-bin\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133212 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-var-lib-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133502 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-89j9l\" (UID: \"9d5e73f9-907c-40b1-a596-b079ded3abfb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133503 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-cni-bin\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133529 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-kubelet\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133530 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-system-cni-dir\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133564 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-cni-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133578 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/afe07c4f-7282-4239-9761-c61ef4153cd9-dbus\") pod \"global-pull-secret-syncer-jjtbq\" (UID: \"afe07c4f-7282-4239-9761-c61ef4153cd9\") " pod="kube-system/global-pull-secret-syncer-jjtbq" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133609 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-system-cni-dir\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133641 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-openvswitch\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133662 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-kubernetes\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133700 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133706 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-kubernetes\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.133665 4971 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133726 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-host\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133771 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-host\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.133805 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert podName:9d5e73f9-907c-40b1-a596-b079ded3abfb nodeName:}" failed. No retries permitted until 2026-11-02 20:14:33.633789684 +0000 UTC m=+1.923446194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-89j9l" (UID: "9d5e73f9-907c-40b1-a596-b079ded3abfb") : secret "networking-console-plugin-cert" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133830 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-conf-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133861 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133895 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133919 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133935 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-multus-conf-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133942 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-etc-selinux\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133963 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-hostroot\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.133984 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.134000 4971 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.134009 4971 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6578786844-fgcrv: secret "image-registry-tls" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134033 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-run-netns\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.134066 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls podName:02eded99-edf7-41a0-8866-549252be5fe0 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:33.634052422 +0000 UTC m=+1.923708941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls") pod "image-registry-6578786844-fgcrv" (UID: "02eded99-edf7-41a0-8866-549252be5fe0") : secret "image-registry-tls" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.134127 4971 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.134139 4971 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.134162 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs podName:30d6b618-cc4e-48b8-a7b4-9649a4242aad nodeName:}" failed. No retries permitted until 2026-11-02 20:14:33.634155593 +0000 UTC m=+1.923812099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs") pod "network-metrics-daemon-5kzc7" (UID: "30d6b618-cc4e-48b8-a7b4-9649a4242aad") : secret "metrics-daemon-secret" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134005 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-run-netns\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134191 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-systemd\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.134206 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls podName:283c8ed3-38ac-4a45-82d0-ec5bd9245376 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:33.634190882 +0000 UTC m=+1.923847392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls") pod "dns-default-8lr5b" (UID: "283c8ed3-38ac-4a45-82d0-ec5bd9245376") : secret "dns-default-metrics-tls" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134220 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-systemd\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134236 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-multus-certs\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134245 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-hostroot\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134261 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-ovn\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134281 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-etc-selinux\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134283 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-log-socket\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134310 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134307 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134335 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-registration-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134340 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134357 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-sys\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134376 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-system-cni-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134381 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-multus-certs\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134391 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-cnibin\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134407 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134412 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-run-ovn\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134422 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-sys-fs\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134443 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-log-socket\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134446 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysctl-conf\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134479 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134503 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-cni-multus\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134521 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-sysctl-conf\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134526 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-modprobe-d\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134547 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-run\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134552 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-registration-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134572 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-node-log\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134588 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-system-cni-dir\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134597 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134617 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-cnibin\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134621 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134644 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-socket-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.134659 4971 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134666 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-kubelet\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.134688 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls podName:6fea26b0-2cef-4683-a20f-620ef0880bfc nodeName:}" failed. No retries permitted until 2026-11-02 20:14:33.634676626 +0000 UTC m=+1.924333134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ndzpm" (UID: "6fea26b0-2cef-4683-a20f-620ef0880bfc") : secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134704 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-lib-modules\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134709 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-host-kubelet\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134742 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-etc-kubernetes\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134755 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-sys-fs\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134771 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c868547-5957-44d5-99d3-4aefc664abb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134783 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-var-lib-cni-multus\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134804 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-netns\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134835 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68a10036-5c78-4b01-b06d-075a727a5d0b-host-slash\") pod \"iptables-alerter-l54nk\" (UID: \"68a10036-5c78-4b01-b06d-075a727a5d0b\") " pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134848 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-etc-modprobe-d\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134860 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134875 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-run\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134879 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-lib-modules\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134883 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-host\") pod \"node-ca-7pl28\" (UID: \"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd\") " pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134899 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-etc-kubernetes\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134909 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-device-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134914 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd-host\") pod \"node-ca-7pl28\" (UID: \"c2a9f726-c51f-4668-a4b3-5eb8ca1df1cd\") " pod="openshift-image-registry/node-ca-7pl28" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134925 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134942 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/39e68d28-771c-48e8-ba38-f8aad80a6bf9-hosts-file\") pod \"node-resolver-26nmv\" (UID: \"39e68d28-771c-48e8-ba38-f8aad80a6bf9\") " pod="openshift-dns/node-resolver-26nmv" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134949 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68a10036-5c78-4b01-b06d-075a727a5d0b-host-slash\") pod \"iptables-alerter-l54nk\" (UID: \"68a10036-5c78-4b01-b06d-075a727a5d0b\") " pod="openshift-network-operator/iptables-alerter-l54nk" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134925 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9fec747e-0efa-40f5-afaf-d45d820700b1-host-run-netns\") pod \"multus-gxrvt\" (UID: \"9fec747e-0efa-40f5-afaf-d45d820700b1\") " pod="openshift-multus/multus-gxrvt" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134989 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcc543bf-8ffd-488d-999a-2b8da64e9c31-sys\") pod \"tuned-fv8rs\" (UID: \"bcc543bf-8ffd-488d-999a-2b8da64e9c31\") " pod="openshift-cluster-node-tuning-operator/tuned-fv8rs" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.134999 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-socket-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.134765 4971 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.135021 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e96f673-1cf6-48f1-a55d-347662c75b16-device-dir\") pod \"aws-ebs-csi-driver-node-bhhcm\" (UID: \"1e96f673-1cf6-48f1-a55d-347662c75b16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bhhcm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.135030 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64f527e7-dcc8-49be-8c15-a078a7aed186-node-log\") pod \"ovnkube-node-9ngnp\" (UID: \"64f527e7-dcc8-49be-8c15-a078a7aed186\") " pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.135038 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls podName:2c868547-5957-44d5-99d3-4aefc664abb1 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:33.635027935 +0000 UTC m=+1.924684441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l4q8k" (UID: "2c868547-5957-44d5-99d3-4aefc664abb1") : secret "insights-runtime-extractor-tls" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.134813 4971 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.135061 4971 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.135065 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c868547-5957-44d5-99d3-4aefc664abb1-kube-rbac-proxy-cm podName:2c868547-5957-44d5-99d3-4aefc664abb1 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:33.635058632 +0000 UTC m=+1.924715137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c868547-5957-44d5-99d3-4aefc664abb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l4q8k" (UID: "2c868547-5957-44d5-99d3-4aefc664abb1") : configmap "kube-rbac-proxy" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.134972 4971 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.135069 4971 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d646d5849-btrvm: secret "image-registry-tls" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.135072 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/39e68d28-771c-48e8-ba38-f8aad80a6bf9-hosts-file\") pod \"node-resolver-26nmv\" (UID: \"39e68d28-771c-48e8-ba38-f8aad80a6bf9\") " pod="openshift-dns/node-resolver-26nmv" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.135108 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls podName:8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed nodeName:}" failed. No retries permitted until 2026-11-02 20:14:33.635098563 +0000 UTC m=+1.924755068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-77d6r" (UID: "8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed") : secret "samples-operator-tls" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.135122 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls podName:9cb81e57-c7dc-4447-959f-d5496ffee976 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:33.635116946 +0000 UTC m=+1.924773452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls") pod "image-registry-5d646d5849-btrvm" (UID: "9cb81e57-c7dc-4447-959f-d5496ffee976") : secret "image-registry-tls" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.135140 4971 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.135150 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a138b9a8-1f14-42a3-97f6-9e2612f83470-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bvfmm\" (UID: \"a138b9a8-1f14-42a3-97f6-9e2612f83470\") " pod="openshift-multus/multus-additional-cni-plugins-bvfmm" Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.135162 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs podName:cb6295a6-f31e-48a3-b4df-7d872a961161 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:33.63515373 +0000 UTC m=+1.924810236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs") pod "router-default-f6f5bd44c-4tksg" (UID: "cb6295a6-f31e-48a3-b4df-7d872a961161") : secret "router-metrics-certs-default" not found Nov 02 20:14:33.135650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.135578 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb6295a6-f31e-48a3-b4df-7d872a961161-service-ca-bundle\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:33.258924 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.258899 4971 scope.go:122] "RemoveContainer" containerID="05b60e41e161918b16b6c84ef6c992e2704d297642ef014aee8dfd11bbdfb89b" Nov 02 20:14:33.465071 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.465048 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-278nr_99e51c7f-420e-4709-b098-2ac9f0eaaedb/serve-healthcheck-canary/0.log" Nov 02 20:14:33.638681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.638626 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-89j9l\" (UID: \"9d5e73f9-907c-40b1-a596-b079ded3abfb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:33.638681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.638666 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:33.638681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.638682 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.638913 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.638697 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:33.638913 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.638791 4971 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 20:14:33.638913 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.638824 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:33.638913 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.638850 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs podName:30d6b618-cc4e-48b8-a7b4-9649a4242aad nodeName:}" failed. No retries permitted until 2026-11-02 20:14:34.638831954 +0000 UTC m=+2.928488479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs") pod "network-metrics-daemon-5kzc7" (UID: "30d6b618-cc4e-48b8-a7b4-9649a4242aad") : secret "metrics-daemon-secret" not found Nov 02 20:14:33.638913 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.638876 4971 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 20:14:33.638913 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.638900 4971 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:33.639233 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.638927 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls podName:283c8ed3-38ac-4a45-82d0-ec5bd9245376 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:34.638911973 +0000 UTC m=+2.928568481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls") pod "dns-default-8lr5b" (UID: "283c8ed3-38ac-4a45-82d0-ec5bd9245376") : secret "dns-default-metrics-tls" not found Nov 02 20:14:33.639233 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.638899 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:33.639233 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.638945 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls podName:6fea26b0-2cef-4683-a20f-620ef0880bfc nodeName:}" failed. No retries permitted until 2026-11-02 20:14:34.638936209 +0000 UTC m=+2.928592718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ndzpm" (UID: "6fea26b0-2cef-4683-a20f-620ef0880bfc") : secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:33.639233 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.638973 4971 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 20:14:33.639233 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.638985 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:33.639233 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.639002 4971 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 20:14:33.639233 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.639016 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls podName:2c868547-5957-44d5-99d3-4aefc664abb1 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:34.639004388 +0000 UTC m=+2.928660911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l4q8k" (UID: "2c868547-5957-44d5-99d3-4aefc664abb1") : secret "insights-runtime-extractor-tls" not found Nov 02 20:14:33.639233 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.639048 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c868547-5957-44d5-99d3-4aefc664abb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:33.639233 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.639055 4971 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 20:14:33.639233 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.639109 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:33.639233 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.639119 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs podName:cb6295a6-f31e-48a3-b4df-7d872a961161 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:34.639105457 +0000 UTC m=+2.928761963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs") pod "router-default-f6f5bd44c-4tksg" (UID: "cb6295a6-f31e-48a3-b4df-7d872a961161") : secret "router-metrics-certs-default" not found Nov 02 20:14:33.639233 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.639156 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.639720 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.639503 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c868547-5957-44d5-99d3-4aefc664abb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:33.641229 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.641214 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls\") pod \"image-registry-6578786844-fgcrv\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.641642 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.641626 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls\") pod \"image-registry-5d646d5849-btrvm\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.641946 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.641926 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d5e73f9-907c-40b1-a596-b079ded3abfb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-89j9l\" (UID: \"9d5e73f9-907c-40b1-a596-b079ded3abfb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:33.642458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.642444 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-77d6r\" (UID: \"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:33.858322 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.858283 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" Nov 02 20:14:33.859410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.859390 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:33.860469 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.860453 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" Nov 02 20:14:33.872411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.872385 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:33.930604 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.924611 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:14:33.930604 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.925103 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/1.log" Nov 02 20:14:33.930604 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.925137 4971 generic.go:356] "Generic (PLEG): container finished" podID="a6fea55b-74c2-40b1-9952-cd767e448317" containerID="8d53479c19a86baf48848b0b857da9bc71961f7493ecd1c0bd5c0ba343751d66" exitCode=255 Nov 02 20:14:33.930604 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.926002 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" event={"ID":"a6fea55b-74c2-40b1-9952-cd767e448317","Type":"ContainerDied","Data":"8d53479c19a86baf48848b0b857da9bc71961f7493ecd1c0bd5c0ba343751d66"} Nov 02 20:14:33.930604 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.926055 4971 scope.go:122] "RemoveContainer" containerID="05b60e41e161918b16b6c84ef6c992e2704d297642ef014aee8dfd11bbdfb89b" Nov 02 20:14:33.930604 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:33.926842 4971 scope.go:122] "RemoveContainer" containerID="8d53479c19a86baf48848b0b857da9bc71961f7493ecd1c0bd5c0ba343751d66" Nov 02 20:14:33.930604 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:33.927017 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2kxv2_openshift-console-operator(a6fea55b-74c2-40b1-9952-cd767e448317)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" podUID="a6fea55b-74c2-40b1-9952-cd767e448317" Nov 02 20:14:34.240218 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.240155 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r"] Nov 02 20:14:34.290363 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.290340 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6578786844-fgcrv"] Nov 02 20:14:34.324023 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:34.323997 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02eded99_edf7_41a0_8866_549252be5fe0.slice/crio-9f32bbd14e766f8b66dcd4613e4ebd480a8b133714cb69d6cf69ff30a8af3de5 WatchSource:0}: Error finding container 9f32bbd14e766f8b66dcd4613e4ebd480a8b133714cb69d6cf69ff30a8af3de5: Status 404 returned error can't find the container with id 9f32bbd14e766f8b66dcd4613e4ebd480a8b133714cb69d6cf69ff30a8af3de5 Nov 02 20:14:34.339406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.339386 4971 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:14:34.339504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.339453 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d646d5849-btrvm"] Nov 02 20:14:34.343412 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:34.343388 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb81e57_c7dc_4447_959f_d5496ffee976.slice/crio-05548d62c130fd64f54c8c1b9903be3863527c077ee4ca5f98becadcb2948dc9 WatchSource:0}: Error finding container 05548d62c130fd64f54c8c1b9903be3863527c077ee4ca5f98becadcb2948dc9: Status 404 returned error can't find the container with id 05548d62c130fd64f54c8c1b9903be3863527c077ee4ca5f98becadcb2948dc9 Nov 02 20:14:34.349562 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.349542 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-89j9l"] Nov 02 20:14:34.353010 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:34.352984 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5e73f9_907c_40b1_a596_b079ded3abfb.slice/crio-45e6df44c0e1506d724e8e8a41c7f36d2aba35a69f6361247caf3fdc06efd44e WatchSource:0}: Error finding container 45e6df44c0e1506d724e8e8a41c7f36d2aba35a69f6361247caf3fdc06efd44e: Status 404 returned error can't find the container with id 45e6df44c0e1506d724e8e8a41c7f36d2aba35a69f6361247caf3fdc06efd44e Nov 02 20:14:34.649682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.649650 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:34.649682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.649684 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:34.649966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.649711 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:34.649966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.649786 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:34.649966 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:34.649802 4971 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:34.649966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.649836 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:34.649966 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:34.649860 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls podName:6fea26b0-2cef-4683-a20f-620ef0880bfc nodeName:}" failed. No retries permitted until 2026-11-02 20:14:36.649843416 +0000 UTC m=+4.939499942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ndzpm" (UID: "6fea26b0-2cef-4683-a20f-620ef0880bfc") : secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:34.649966 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:34.649868 4971 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 20:14:34.649966 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:34.649891 4971 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 20:14:34.649966 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:34.649904 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls podName:2c868547-5957-44d5-99d3-4aefc664abb1 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:36.649894399 +0000 UTC m=+4.939550908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l4q8k" (UID: "2c868547-5957-44d5-99d3-4aefc664abb1") : secret "insights-runtime-extractor-tls" not found Nov 02 20:14:34.649966 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:34.649955 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls podName:283c8ed3-38ac-4a45-82d0-ec5bd9245376 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:36.649936654 +0000 UTC m=+4.939593183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls") pod "dns-default-8lr5b" (UID: "283c8ed3-38ac-4a45-82d0-ec5bd9245376") : secret "dns-default-metrics-tls" not found Nov 02 20:14:34.652304 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.652276 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d6b618-cc4e-48b8-a7b4-9649a4242aad-metrics-certs\") pod \"network-metrics-daemon-5kzc7\" (UID: \"30d6b618-cc4e-48b8-a7b4-9649a4242aad\") " pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:34.652445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.652338 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6295a6-f31e-48a3-b4df-7d872a961161-metrics-certs\") pod \"router-default-f6f5bd44c-4tksg\" (UID: \"cb6295a6-f31e-48a3-b4df-7d872a961161\") " pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:34.758787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.758751 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:34.765854 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.765825 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kzc7" Nov 02 20:14:34.930397 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.930332 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" event={"ID":"9d5e73f9-907c-40b1-a596-b079ded3abfb","Type":"ContainerStarted","Data":"45e6df44c0e1506d724e8e8a41c7f36d2aba35a69f6361247caf3fdc06efd44e"} Nov 02 20:14:34.932063 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.932033 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6578786844-fgcrv" event={"ID":"02eded99-edf7-41a0-8866-549252be5fe0","Type":"ContainerStarted","Data":"d2a3afe03bb144cd81b315511993bbecb17e5e3b4e914cbdbb2c2af5a0621b1b"} Nov 02 20:14:34.932201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.932106 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6578786844-fgcrv" event={"ID":"02eded99-edf7-41a0-8866-549252be5fe0","Type":"ContainerStarted","Data":"9f32bbd14e766f8b66dcd4613e4ebd480a8b133714cb69d6cf69ff30a8af3de5"} Nov 02 20:14:34.933699 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.933673 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" event={"ID":"9cb81e57-c7dc-4447-959f-d5496ffee976","Type":"ContainerStarted","Data":"f514d4af192b5d56100c0ea6e339b5a63187275f0e34251c0173ba52fced8848"} Nov 02 20:14:34.933822 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.933711 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" event={"ID":"9cb81e57-c7dc-4447-959f-d5496ffee976","Type":"ContainerStarted","Data":"05548d62c130fd64f54c8c1b9903be3863527c077ee4ca5f98becadcb2948dc9"} Nov 02 20:14:34.935320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.935304 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:14:34.936542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:34.936518 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" event={"ID":"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed","Type":"ContainerStarted","Data":"7c3e252022c6f93b9ab0720c63789a60503ab168b3d4b56d9cebfd74411d502c"} Nov 02 20:14:35.152471 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:35.152442 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f6f5bd44c-4tksg"] Nov 02 20:14:35.155163 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:35.155138 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6295a6_f31e_48a3_b4df_7d872a961161.slice/crio-87ed47ac1e0ac3a1f011b2cdb28d955a32102b560e2545eb5ef05ceb413f93f5 WatchSource:0}: Error finding container 87ed47ac1e0ac3a1f011b2cdb28d955a32102b560e2545eb5ef05ceb413f93f5: Status 404 returned error can't find the container with id 87ed47ac1e0ac3a1f011b2cdb28d955a32102b560e2545eb5ef05ceb413f93f5 Nov 02 20:14:35.187709 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:35.187686 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5kzc7"] Nov 02 20:14:35.191340 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:35.191312 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d6b618_cc4e_48b8_a7b4_9649a4242aad.slice/crio-b7cdcb7929406be585dd226824c02de39bc5d34c45cfa62d08dfb77194d68bd9 WatchSource:0}: Error finding container b7cdcb7929406be585dd226824c02de39bc5d34c45cfa62d08dfb77194d68bd9: Status 404 returned error can't find the container with id b7cdcb7929406be585dd226824c02de39bc5d34c45cfa62d08dfb77194d68bd9 Nov 02 20:14:35.484434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:35.484357 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:35.673121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:35.673065 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:14:35.941800 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:35.941770 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kzc7" event={"ID":"30d6b618-cc4e-48b8-a7b4-9649a4242aad","Type":"ContainerStarted","Data":"b7cdcb7929406be585dd226824c02de39bc5d34c45cfa62d08dfb77194d68bd9"} Nov 02 20:14:35.943785 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:35.943748 4971 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 20:14:35.943915 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:35.943754 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f6f5bd44c-4tksg" event={"ID":"cb6295a6-f31e-48a3-b4df-7d872a961161","Type":"ContainerStarted","Data":"6e91f51143032072ddea54ec7563ec219bcb387d01798e2f6c7f3382d905977c"} Nov 02 20:14:35.943987 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:35.943930 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f6f5bd44c-4tksg" event={"ID":"cb6295a6-f31e-48a3-b4df-7d872a961161","Type":"ContainerStarted","Data":"87ed47ac1e0ac3a1f011b2cdb28d955a32102b560e2545eb5ef05ceb413f93f5"} Nov 02 20:14:36.669503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:36.669463 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:36.669689 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:36.669522 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:36.669689 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:36.669615 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:36.669807 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:36.669727 4971 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 20:14:36.669807 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:36.669789 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls podName:283c8ed3-38ac-4a45-82d0-ec5bd9245376 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:40.669774635 +0000 UTC m=+8.959431142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls") pod "dns-default-8lr5b" (UID: "283c8ed3-38ac-4a45-82d0-ec5bd9245376") : secret "dns-default-metrics-tls" not found Nov 02 20:14:36.670227 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:36.670207 4971 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:36.670334 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:36.670255 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls podName:6fea26b0-2cef-4683-a20f-620ef0880bfc nodeName:}" failed. No retries permitted until 2026-11-02 20:14:40.670243849 +0000 UTC m=+8.959900355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ndzpm" (UID: "6fea26b0-2cef-4683-a20f-620ef0880bfc") : secret "cluster-monitoring-operator-tls" not found Nov 02 20:14:36.670334 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:36.670311 4971 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 20:14:36.670417 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:36.670344 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls podName:2c868547-5957-44d5-99d3-4aefc664abb1 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:40.670333139 +0000 UTC m=+8.959989646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l4q8k" (UID: "2c868547-5957-44d5-99d3-4aefc664abb1") : secret "insights-runtime-extractor-tls" not found Nov 02 20:14:36.758965 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:36.758935 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:36.761596 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:36.761571 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:36.950933 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:36.950855 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-89j9l" event={"ID":"9d5e73f9-907c-40b1-a596-b079ded3abfb","Type":"ContainerStarted","Data":"0b1f2e97c11b5ed00eecc2608f93cf72a7bb2b81a58823fbcdc35b1a527dd9b1"} Nov 02 20:14:37.957765 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:37.957729 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kzc7" event={"ID":"30d6b618-cc4e-48b8-a7b4-9649a4242aad","Type":"ContainerStarted","Data":"2ff74e57b66984d5aa7e5d10ab6936cd94790403d2e49d96a6e515e4db7cb480"} Nov 02 20:14:37.958230 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:37.957790 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kzc7" event={"ID":"30d6b618-cc4e-48b8-a7b4-9649a4242aad","Type":"ContainerStarted","Data":"d0a2d2c03fc16d91c986eaeb35cebb7d1889abc82b5a81be545dab2465b73b0c"} Nov 02 20:14:37.960050 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:37.960020 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" event={"ID":"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed","Type":"ContainerStarted","Data":"938fe5e6503810675e90bbe6e4b694ec4c107703088e1199ce82eda8b3d52308"} Nov 02 20:14:37.960050 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:37.960047 4971 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 20:14:37.960311 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:37.960067 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-77d6r" event={"ID":"8f2b73e9-e8ed-4b00-9dfe-8b579eb020ed","Type":"ContainerStarted","Data":"1d4943e502bd5fe6eea1828abfb6e89a825e89c7ecb81949ff4fb01221dfbba6"} Nov 02 20:14:37.966676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:37.966656 4971 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:37.966922 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:37.966908 4971 scope.go:122] "RemoveContainer" containerID="8d53479c19a86baf48848b0b857da9bc71961f7493ecd1c0bd5c0ba343751d66" Nov 02 20:14:37.967054 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:37.967037 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2kxv2_openshift-console-operator(a6fea55b-74c2-40b1-9952-cd767e448317)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" podUID="a6fea55b-74c2-40b1-9952-cd767e448317" Nov 02 20:14:38.297042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:38.296961 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:38.298575 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:38.298551 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-f6f5bd44c-4tksg" Nov 02 20:14:38.837867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:38.837833 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:38.964162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:38.964134 4971 scope.go:122] "RemoveContainer" containerID="8d53479c19a86baf48848b0b857da9bc71961f7493ecd1c0bd5c0ba343751d66" Nov 02 20:14:38.964606 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:38.964337 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2kxv2_openshift-console-operator(a6fea55b-74c2-40b1-9952-cd767e448317)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" podUID="a6fea55b-74c2-40b1-9952-cd767e448317" Nov 02 20:14:39.720309 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:39.720278 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jbk5q" Nov 02 20:14:40.703136 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:40.703095 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:40.703561 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:40.703146 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:40.703561 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:40.703200 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:40.703561 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:40.703290 4971 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 20:14:40.703561 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:40.703289 4971 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 20:14:40.703561 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:40.703369 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls podName:283c8ed3-38ac-4a45-82d0-ec5bd9245376 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:48.703355656 +0000 UTC m=+16.993012162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls") pod "dns-default-8lr5b" (UID: "283c8ed3-38ac-4a45-82d0-ec5bd9245376") : secret "dns-default-metrics-tls" not found Nov 02 20:14:40.703561 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:14:40.703382 4971 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls podName:2c868547-5957-44d5-99d3-4aefc664abb1 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:48.703376994 +0000 UTC m=+16.993033501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l4q8k" (UID: "2c868547-5957-44d5-99d3-4aefc664abb1") : secret "insights-runtime-extractor-tls" not found Nov 02 20:14:40.705757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:40.705730 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6fea26b0-2cef-4683-a20f-620ef0880bfc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ndzpm\" (UID: \"6fea26b0-2cef-4683-a20f-620ef0880bfc\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:40.759379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:40.759350 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" Nov 02 20:14:40.995158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:40.995097 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:14:41.151655 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:41.151629 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm"] Nov 02 20:14:41.157638 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:41.157611 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fea26b0_2cef_4683_a20f_620ef0880bfc.slice/crio-74d9df454513d234a9b2fb7ab08b12689e1ecb01cd91824ef6b8a5c2a60b7c37 WatchSource:0}: Error finding container 74d9df454513d234a9b2fb7ab08b12689e1ecb01cd91824ef6b8a5c2a60b7c37: Status 404 returned error can't find the container with id 74d9df454513d234a9b2fb7ab08b12689e1ecb01cd91824ef6b8a5c2a60b7c37 Nov 02 20:14:41.387170 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:41.387147 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:14:41.975001 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:41.974962 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" event={"ID":"6fea26b0-2cef-4683-a20f-620ef0880bfc","Type":"ContainerStarted","Data":"74d9df454513d234a9b2fb7ab08b12689e1ecb01cd91824ef6b8a5c2a60b7c37"} Nov 02 20:14:42.222535 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:42.222509 4971 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-120.us-west-2.compute.internal" event="NodeReady" Nov 02 20:14:42.838225 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:42.838198 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:42.838444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:42.838431 4971 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 20:14:43.707902 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:43.707867 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:43.994549 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:43.994477 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:45.303801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:45.303769 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9ngnp" Nov 02 20:14:45.859359 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:45.859328 4971 scope.go:122] "RemoveContainer" containerID="8d53479c19a86baf48848b0b857da9bc71961f7493ecd1c0bd5c0ba343751d66" Nov 02 20:14:45.888439 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:45.888404 4971 patch_prober.go:29] interesting pod/image-registry-5d646d5849-btrvm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 20:14:45.888536 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:45.888507 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" podUID="9cb81e57-c7dc-4447-959f-d5496ffee976" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:14:45.990322 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:45.990287 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ndzpm" event={"ID":"6fea26b0-2cef-4683-a20f-620ef0880bfc","Type":"ContainerStarted","Data":"c955dd7acb388fce924da40bd34ddbe277ccfda18201707507e8df86a8092427"} Nov 02 20:14:45.992158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:45.992137 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:14:45.992280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:45.992187 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" event={"ID":"a6fea55b-74c2-40b1-9952-cd767e448317","Type":"ContainerStarted","Data":"49aad6c1c9d7bcd1265159a650b1574bfb1c5ddbcbd6159c0ce47dd4bbd271d2"} Nov 02 20:14:45.992435 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:45.992418 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:45.993118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:45.993100 4971 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-2kxv2 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body= Nov 02 20:14:45.993160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:45.993136 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" podUID="a6fea55b-74c2-40b1-9952-cd767e448317" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" Nov 02 20:14:47.000012 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:46.999983 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2kxv2" Nov 02 20:14:47.063989 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:47.063964 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:47.304321 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:47.304240 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:47.549581 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:47.549539 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:47.869935 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:47.869904 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:48.110501 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:48.110468 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:48.362320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:48.362283 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:48.723449 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:48.723385 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:48.767300 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:48.767270 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:48.767433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:48.767346 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:48.769801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:48.769769 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c868547-5957-44d5-99d3-4aefc664abb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l4q8k\" (UID: \"2c868547-5957-44d5-99d3-4aefc664abb1\") " pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:48.769919 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:48.769812 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/283c8ed3-38ac-4a45-82d0-ec5bd9245376-metrics-tls\") pod \"dns-default-8lr5b\" (UID: \"283c8ed3-38ac-4a45-82d0-ec5bd9245376\") " pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:48.873173 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:48.873146 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:48.875296 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:48.875276 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l4q8k" Nov 02 20:14:48.965573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:48.965534 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:49.339381 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:49.339355 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8lr5b"] Nov 02 20:14:49.341911 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:49.341886 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283c8ed3_38ac_4a45_82d0_ec5bd9245376.slice/crio-5c52a56cd63086f3ce6f57df9fcf24c6c415b9ff7ab11531bfd2fc15e076d6c1 WatchSource:0}: Error finding container 5c52a56cd63086f3ce6f57df9fcf24c6c415b9ff7ab11531bfd2fc15e076d6c1: Status 404 returned error can't find the container with id 5c52a56cd63086f3ce6f57df9fcf24c6c415b9ff7ab11531bfd2fc15e076d6c1 Nov 02 20:14:49.376131 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:49.376096 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c868547_5957_44d5_99d3_4aefc664abb1.slice/crio-d98e76d7549038eedeb2a577726b9c0402749d1a8d28270f91bbb032547f6cad WatchSource:0}: Error finding container d98e76d7549038eedeb2a577726b9c0402749d1a8d28270f91bbb032547f6cad: Status 404 returned error can't find the container with id d98e76d7549038eedeb2a577726b9c0402749d1a8d28270f91bbb032547f6cad Nov 02 20:14:49.385842 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:49.385805 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l4q8k"] Nov 02 20:14:50.011436 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:50.011402 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l4q8k" event={"ID":"2c868547-5957-44d5-99d3-4aefc664abb1","Type":"ContainerStarted","Data":"fcf2afc205b85e8fe776775039847e686dab1e50092802fefa344c45d8254caa"} Nov 02 20:14:50.011630 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:50.011464 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l4q8k" event={"ID":"2c868547-5957-44d5-99d3-4aefc664abb1","Type":"ContainerStarted","Data":"d98e76d7549038eedeb2a577726b9c0402749d1a8d28270f91bbb032547f6cad"} Nov 02 20:14:50.012544 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:50.012515 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8lr5b" event={"ID":"283c8ed3-38ac-4a45-82d0-ec5bd9245376","Type":"ContainerStarted","Data":"5c52a56cd63086f3ce6f57df9fcf24c6c415b9ff7ab11531bfd2fc15e076d6c1"} Nov 02 20:14:51.392851 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:51.392816 4971 patch_prober.go:29] interesting pod/image-registry-5d646d5849-btrvm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 20:14:51.393257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:51.392875 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" podUID="9cb81e57-c7dc-4447-959f-d5496ffee976" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:14:51.395971 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:51.395948 4971 patch_prober.go:29] interesting pod/image-registry-6578786844-fgcrv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 20:14:51.396120 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:51.395984 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6578786844-fgcrv" podUID="02eded99-edf7-41a0-8866-549252be5fe0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:14:52.022138 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:52.022107 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l4q8k" event={"ID":"2c868547-5957-44d5-99d3-4aefc664abb1","Type":"ContainerStarted","Data":"e03a0d927750803055a1abf871bf128e9444b710cb211a175c7c4ba0a8256e15"} Nov 02 20:14:53.314106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.314053 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b94zk"] Nov 02 20:14:53.316689 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.316667 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b94zk" Nov 02 20:14:53.341999 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.341977 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vpmcb"] Nov 02 20:14:53.344404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.344385 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vpmcb" Nov 02 20:14:53.361298 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.361276 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-wfwqz"] Nov 02 20:14:53.364155 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.363592 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-wfwqz" Nov 02 20:14:53.376707 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.376689 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6578786844-fgcrv"] Nov 02 20:14:53.381216 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.381181 4971 patch_prober.go:29] interesting pod/image-registry-6578786844-fgcrv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 20:14:53.381308 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.381228 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6578786844-fgcrv" podUID="02eded99-edf7-41a0-8866-549252be5fe0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:14:53.386292 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.386269 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8vvv2"] Nov 02 20:14:53.388523 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.388509 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vvv2" Nov 02 20:14:53.399127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.399109 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lqm9f\"" Nov 02 20:14:53.399219 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.399107 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 20:14:53.401370 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.401350 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gch82\" (UniqueName: \"kubernetes.io/projected/d9fa6f53-0a85-4f9d-8d9c-4498923df670-kube-api-access-gch82\") pod \"migrator-66df5d496b-vpmcb\" (UID: \"d9fa6f53-0a85-4f9d-8d9c-4498923df670\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vpmcb" Nov 02 20:14:53.401467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.401420 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1f6f9164-5e01-4400-a606-adf3c5016a61-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-b94zk\" (UID: \"1f6f9164-5e01-4400-a606-adf3c5016a61\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b94zk" Nov 02 20:14:53.406900 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.406883 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 20:14:53.407038 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.407020 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 20:14:53.415481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.415462 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-dc7dd847c-sl2nq"] Nov 02 20:14:53.417660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.417645 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.420174 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.420156 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 20:14:53.422444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.422427 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 20:14:53.436823 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.436802 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b94zk"] Nov 02 20:14:53.437696 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.437671 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vpmcb"] Nov 02 20:14:53.440416 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.440369 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-wfwqz"] Nov 02 20:14:53.472501 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.472476 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cvg7r\"" Nov 02 20:14:53.472501 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.472481 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-kfwfm\"" Nov 02 20:14:53.489609 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.489588 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dc7dd847c-sl2nq"] Nov 02 20:14:53.492509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.492486 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:53.495719 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.495697 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8vvv2"] Nov 02 20:14:53.502408 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.502380 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69b4285c-bc77-47f9-ae3e-33210d640283-tmp\") pod \"downloads-6dd789c8b7-wfwqz\" (UID: \"69b4285c-bc77-47f9-ae3e-33210d640283\") " pod="openshift-console/downloads-6dd789c8b7-wfwqz" Nov 02 20:14:53.502517 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.502418 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjqn4\" (UniqueName: \"kubernetes.io/projected/360a57b3-cbd3-4620-a9b0-222d457a085c-kube-api-access-jjqn4\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.502517 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.502435 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w64bc\" (UniqueName: \"kubernetes.io/projected/69b4285c-bc77-47f9-ae3e-33210d640283-kube-api-access-w64bc\") pod \"downloads-6dd789c8b7-wfwqz\" (UID: \"69b4285c-bc77-47f9-ae3e-33210d640283\") " pod="openshift-console/downloads-6dd789c8b7-wfwqz" Nov 02 20:14:53.502597 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.502581 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gch82\" (UniqueName: \"kubernetes.io/projected/d9fa6f53-0a85-4f9d-8d9c-4498923df670-kube-api-access-gch82\") pod \"migrator-66df5d496b-vpmcb\" (UID: \"d9fa6f53-0a85-4f9d-8d9c-4498923df670\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vpmcb" Nov 02 20:14:53.502641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.502622 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nffqb\" (UniqueName: \"kubernetes.io/projected/b16b8d53-43f2-47ba-8a29-8e6bccd085b4-kube-api-access-nffqb\") pod \"migrator-66df5d496b-8vvv2\" (UID: \"b16b8d53-43f2-47ba-8a29-8e6bccd085b4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vvv2" Nov 02 20:14:53.502678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.502662 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/360a57b3-cbd3-4620-a9b0-222d457a085c-registry-certificates\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.502728 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.502702 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/360a57b3-cbd3-4620-a9b0-222d457a085c-image-registry-private-configuration\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.502820 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.502802 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/360a57b3-cbd3-4620-a9b0-222d457a085c-trusted-ca\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.502879 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.502836 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/360a57b3-cbd3-4620-a9b0-222d457a085c-installation-pull-secrets\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.502935 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.502902 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/360a57b3-cbd3-4620-a9b0-222d457a085c-bound-sa-token\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.502991 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.502966 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1f6f9164-5e01-4400-a606-adf3c5016a61-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-b94zk\" (UID: \"1f6f9164-5e01-4400-a606-adf3c5016a61\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b94zk" Nov 02 20:14:53.503047 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.503030 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/360a57b3-cbd3-4620-a9b0-222d457a085c-ca-trust-extracted\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.503137 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.503104 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/360a57b3-cbd3-4620-a9b0-222d457a085c-registry-tls\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.506433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.506407 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1f6f9164-5e01-4400-a606-adf3c5016a61-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-b94zk\" (UID: \"1f6f9164-5e01-4400-a606-adf3c5016a61\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b94zk" Nov 02 20:14:53.560479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.560416 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gch82\" (UniqueName: \"kubernetes.io/projected/d9fa6f53-0a85-4f9d-8d9c-4498923df670-kube-api-access-gch82\") pod \"migrator-66df5d496b-vpmcb\" (UID: \"d9fa6f53-0a85-4f9d-8d9c-4498923df670\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vpmcb" Nov 02 20:14:53.604295 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.604269 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjqn4\" (UniqueName: \"kubernetes.io/projected/360a57b3-cbd3-4620-a9b0-222d457a085c-kube-api-access-jjqn4\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.604442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.604299 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w64bc\" (UniqueName: \"kubernetes.io/projected/69b4285c-bc77-47f9-ae3e-33210d640283-kube-api-access-w64bc\") pod \"downloads-6dd789c8b7-wfwqz\" (UID: \"69b4285c-bc77-47f9-ae3e-33210d640283\") " pod="openshift-console/downloads-6dd789c8b7-wfwqz" Nov 02 20:14:53.604442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.604326 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nffqb\" (UniqueName: \"kubernetes.io/projected/b16b8d53-43f2-47ba-8a29-8e6bccd085b4-kube-api-access-nffqb\") pod \"migrator-66df5d496b-8vvv2\" (UID: \"b16b8d53-43f2-47ba-8a29-8e6bccd085b4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vvv2" Nov 02 20:14:53.604442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.604350 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/360a57b3-cbd3-4620-a9b0-222d457a085c-registry-certificates\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.604442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.604387 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/360a57b3-cbd3-4620-a9b0-222d457a085c-image-registry-private-configuration\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.604442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.604430 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/360a57b3-cbd3-4620-a9b0-222d457a085c-trusted-ca\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.604678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.604454 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/360a57b3-cbd3-4620-a9b0-222d457a085c-installation-pull-secrets\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.604678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.604478 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/360a57b3-cbd3-4620-a9b0-222d457a085c-bound-sa-token\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.604678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.604527 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/360a57b3-cbd3-4620-a9b0-222d457a085c-ca-trust-extracted\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.605141 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.604573 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/360a57b3-cbd3-4620-a9b0-222d457a085c-registry-tls\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.605265 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.605203 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69b4285c-bc77-47f9-ae3e-33210d640283-tmp\") pod \"downloads-6dd789c8b7-wfwqz\" (UID: \"69b4285c-bc77-47f9-ae3e-33210d640283\") " pod="openshift-console/downloads-6dd789c8b7-wfwqz" Nov 02 20:14:53.605422 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.605378 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/360a57b3-cbd3-4620-a9b0-222d457a085c-ca-trust-extracted\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.605965 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.605789 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/360a57b3-cbd3-4620-a9b0-222d457a085c-registry-certificates\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.606388 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.606327 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/360a57b3-cbd3-4620-a9b0-222d457a085c-trusted-ca\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.607973 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.607947 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/360a57b3-cbd3-4620-a9b0-222d457a085c-registry-tls\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.610876 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.608295 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69b4285c-bc77-47f9-ae3e-33210d640283-tmp\") pod \"downloads-6dd789c8b7-wfwqz\" (UID: \"69b4285c-bc77-47f9-ae3e-33210d640283\") " pod="openshift-console/downloads-6dd789c8b7-wfwqz" Nov 02 20:14:53.610876 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.609625 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/360a57b3-cbd3-4620-a9b0-222d457a085c-image-registry-private-configuration\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.614198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.614176 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/360a57b3-cbd3-4620-a9b0-222d457a085c-installation-pull-secrets\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.630932 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.630901 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b94zk" Nov 02 20:14:53.656675 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.656655 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vpmcb" Nov 02 20:14:53.663683 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.663659 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nffqb\" (UniqueName: \"kubernetes.io/projected/b16b8d53-43f2-47ba-8a29-8e6bccd085b4-kube-api-access-nffqb\") pod \"migrator-66df5d496b-8vvv2\" (UID: \"b16b8d53-43f2-47ba-8a29-8e6bccd085b4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vvv2" Nov 02 20:14:53.664567 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.664546 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/360a57b3-cbd3-4620-a9b0-222d457a085c-bound-sa-token\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.664652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.664617 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjqn4\" (UniqueName: \"kubernetes.io/projected/360a57b3-cbd3-4620-a9b0-222d457a085c-kube-api-access-jjqn4\") pod \"image-registry-dc7dd847c-sl2nq\" (UID: \"360a57b3-cbd3-4620-a9b0-222d457a085c\") " pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.665121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.665074 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w64bc\" (UniqueName: \"kubernetes.io/projected/69b4285c-bc77-47f9-ae3e-33210d640283-kube-api-access-w64bc\") pod \"downloads-6dd789c8b7-wfwqz\" (UID: \"69b4285c-bc77-47f9-ae3e-33210d640283\") " pod="openshift-console/downloads-6dd789c8b7-wfwqz" Nov 02 20:14:53.674935 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.674913 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-wfwqz" Nov 02 20:14:53.701533 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.698399 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vvv2" Nov 02 20:14:53.729593 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.729568 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:53.740012 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:53.739944 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:54.001887 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:54.001860 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b94zk"] Nov 02 20:14:54.005145 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:54.005115 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6f9164_5e01_4400_a606_adf3c5016a61.slice/crio-cf1b28452344ca6497b3d150351b7c645c6cebabefed070b04a810b909846ca4 WatchSource:0}: Error finding container cf1b28452344ca6497b3d150351b7c645c6cebabefed070b04a810b909846ca4: Status 404 returned error can't find the container with id cf1b28452344ca6497b3d150351b7c645c6cebabefed070b04a810b909846ca4 Nov 02 20:14:54.031400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:54.031374 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b94zk" event={"ID":"1f6f9164-5e01-4400-a606-adf3c5016a61","Type":"ContainerStarted","Data":"cf1b28452344ca6497b3d150351b7c645c6cebabefed070b04a810b909846ca4"} Nov 02 20:14:54.043130 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:54.043108 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:54.048143 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:54.048125 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vpmcb"] Nov 02 20:14:54.050897 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:54.050878 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9fa6f53_0a85_4f9d_8d9c_4498923df670.slice/crio-90f6b1737f2c44768f997c1ed22f27242d073b2db21bbb719e4ee26f27dcdb4c WatchSource:0}: Error finding container 90f6b1737f2c44768f997c1ed22f27242d073b2db21bbb719e4ee26f27dcdb4c: Status 404 returned error can't find the container with id 90f6b1737f2c44768f997c1ed22f27242d073b2db21bbb719e4ee26f27dcdb4c Nov 02 20:14:54.073792 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:54.073758 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-wfwqz"] Nov 02 20:14:54.075648 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:54.075618 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b4285c_bc77_47f9_ae3e_33210d640283.slice/crio-13829900856b3afada18108f215f4f820ae50f559ae231bc348674f6674438b8 WatchSource:0}: Error finding container 13829900856b3afada18108f215f4f820ae50f559ae231bc348674f6674438b8: Status 404 returned error can't find the container with id 13829900856b3afada18108f215f4f820ae50f559ae231bc348674f6674438b8 Nov 02 20:14:54.093700 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:54.093679 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8vvv2"] Nov 02 20:14:54.096061 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:54.096039 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16b8d53_43f2_47ba_8a29_8e6bccd085b4.slice/crio-81d36431124491f2873c5a2f05c000ac3248dc17f568ce0fd2c337d58dadca07 WatchSource:0}: Error finding container 81d36431124491f2873c5a2f05c000ac3248dc17f568ce0fd2c337d58dadca07: Status 404 returned error can't find the container with id 81d36431124491f2873c5a2f05c000ac3248dc17f568ce0fd2c337d58dadca07 Nov 02 20:14:54.138293 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:54.138275 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dc7dd847c-sl2nq"] Nov 02 20:14:54.140352 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:54.140332 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360a57b3_cbd3_4620_a9b0_222d457a085c.slice/crio-0f8005242235928b70ebdc239030a634833ff351f4d25ee5b5bf637c27ead8dc WatchSource:0}: Error finding container 0f8005242235928b70ebdc239030a634833ff351f4d25ee5b5bf637c27ead8dc: Status 404 returned error can't find the container with id 0f8005242235928b70ebdc239030a634833ff351f4d25ee5b5bf637c27ead8dc Nov 02 20:14:54.296793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:54.296764 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:55.038274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:55.038201 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l4q8k" event={"ID":"2c868547-5957-44d5-99d3-4aefc664abb1","Type":"ContainerStarted","Data":"2ccf9064038943467b730f1c11bcc6a0407301bd09004ed94ca08f1f64e66699"} Nov 02 20:14:55.039817 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:55.039792 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" event={"ID":"360a57b3-cbd3-4620-a9b0-222d457a085c","Type":"ContainerStarted","Data":"a505057acee1fb5462818f46325df0a775ca4cbb4e5fcc6e94bb0402d1d5f018"} Nov 02 20:14:55.039947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:55.039834 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" event={"ID":"360a57b3-cbd3-4620-a9b0-222d457a085c","Type":"ContainerStarted","Data":"0f8005242235928b70ebdc239030a634833ff351f4d25ee5b5bf637c27ead8dc"} Nov 02 20:14:55.039947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:55.039854 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:14:55.040968 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:55.040949 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vvv2" event={"ID":"b16b8d53-43f2-47ba-8a29-8e6bccd085b4","Type":"ContainerStarted","Data":"81d36431124491f2873c5a2f05c000ac3248dc17f568ce0fd2c337d58dadca07"} Nov 02 20:14:55.042126 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:55.042105 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-wfwqz" event={"ID":"69b4285c-bc77-47f9-ae3e-33210d640283","Type":"ContainerStarted","Data":"13829900856b3afada18108f215f4f820ae50f559ae231bc348674f6674438b8"} Nov 02 20:14:55.043238 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:55.043216 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vpmcb" event={"ID":"d9fa6f53-0a85-4f9d-8d9c-4498923df670","Type":"ContainerStarted","Data":"90f6b1737f2c44768f997c1ed22f27242d073b2db21bbb719e4ee26f27dcdb4c"} Nov 02 20:14:55.266951 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:55.266871 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" podStartSLOduration=2.266853566 podStartE2EDuration="2.266853566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:55.264412195 +0000 UTC m=+23.554068744" watchObservedRunningTime="2026-11-02 20:14:55.266853566 +0000 UTC m=+23.556510094" Nov 02 20:14:55.886659 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:55.886618 4971 patch_prober.go:29] interesting pod/image-registry-5d646d5849-btrvm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 20:14:55.886814 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:55.886685 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" podUID="9cb81e57-c7dc-4447-959f-d5496ffee976" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:14:56.555672 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:56.555583 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:57.053384 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:57.053356 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vvv2" event={"ID":"b16b8d53-43f2-47ba-8a29-8e6bccd085b4","Type":"ContainerStarted","Data":"cd965aa87d60348175d683a8c29eb686743d11fafcabe6871bf1a5afe7c960b9"} Nov 02 20:14:57.055489 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:57.055462 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8lr5b" event={"ID":"283c8ed3-38ac-4a45-82d0-ec5bd9245376","Type":"ContainerStarted","Data":"15fea1977a0717baa953442b12b301bff827e087fa9b5a69347d411371209883"} Nov 02 20:14:57.055630 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:57.055521 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8lr5b" event={"ID":"283c8ed3-38ac-4a45-82d0-ec5bd9245376","Type":"ContainerStarted","Data":"26771008de74a426b93f9707a9707b3c7c9d54d52e2ad343b3a528d6133bb1f6"} Nov 02 20:14:57.055630 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:57.055573 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8lr5b" Nov 02 20:14:57.209540 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:57.209488 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vvv2" podStartSLOduration=1.557859342 podStartE2EDuration="4.209469448s" totalImagesPullingTime="2.651610106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:57.161383391 +0000 UTC m=+25.451039921" watchObservedRunningTime="2026-11-02 20:14:57.209469448 +0000 UTC m=+25.499125979" Nov 02 20:14:57.525648 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:57.525560 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:57.765595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:57.765565 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:14:58.924434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:58.924403 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-799b587cd-csb9b"] Nov 02 20:14:58.945607 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:58.945576 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:58.983370 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:58.983345 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-799b587cd-csb9b"] Nov 02 20:14:59.028856 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.028835 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 20:14:59.041863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.041835 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 20:14:59.041863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.041835 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 20:14:59.042011 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.041866 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 20:14:59.042011 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.041877 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nm4xn\"" Nov 02 20:14:59.042011 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.041890 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 20:14:59.058283 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.058261 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-service-ca\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.058366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.058298 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-config\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.058366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.058318 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-oauth-config\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.058366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.058339 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-oauth-serving-cert\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.058466 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.058400 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lcsf\" (UniqueName: \"kubernetes.io/projected/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-kube-api-access-8lcsf\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.058499 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.058468 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-tmp\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.058529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.058509 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-serving-cert\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.159054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.158972 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-service-ca\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.159054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.159008 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-config\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.159054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.159025 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-oauth-config\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.159054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.159047 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-oauth-serving-cert\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.159455 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.159072 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lcsf\" (UniqueName: \"kubernetes.io/projected/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-kube-api-access-8lcsf\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.159455 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.159131 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-tmp\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.159455 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.159170 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-serving-cert\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.159822 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.159754 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-tmp\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.160016 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.159826 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-service-ca\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.160016 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.159942 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-oauth-serving-cert\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.160234 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.160161 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-config\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.161689 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.161667 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-oauth-config\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.161892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.161873 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-serving-cert\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.216993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.216940 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lcsf\" (UniqueName: \"kubernetes.io/projected/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-kube-api-access-8lcsf\") pod \"console-799b587cd-csb9b\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.277781 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.277756 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:14:59.676115 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:14:59.676002 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-799b587cd-csb9b"] Nov 02 20:14:59.678055 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:14:59.678031 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d8986b_2bb5_4ad9_88a7_fe81cb1f3183.slice/crio-5be09a529ddfb8e3e5131e840835f24f2937b0bad28eacb184dcf1a20f6d68db WatchSource:0}: Error finding container 5be09a529ddfb8e3e5131e840835f24f2937b0bad28eacb184dcf1a20f6d68db: Status 404 returned error can't find the container with id 5be09a529ddfb8e3e5131e840835f24f2937b0bad28eacb184dcf1a20f6d68db Nov 02 20:15:00.069828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:00.069753 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vpmcb" event={"ID":"d9fa6f53-0a85-4f9d-8d9c-4498923df670","Type":"ContainerStarted","Data":"f1d48b0d3af25d63e16800a4a160c3b646d60b7e0f499a53684600880e3df1d2"} Nov 02 20:15:00.071452 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:00.071427 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799b587cd-csb9b" event={"ID":"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183","Type":"ContainerStarted","Data":"5be09a529ddfb8e3e5131e840835f24f2937b0bad28eacb184dcf1a20f6d68db"} Nov 02 20:15:00.179137 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:00.179064 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vpmcb" podStartSLOduration=1.417509193 podStartE2EDuration="7.179052691s" totalImagesPullingTime="5.761543498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:00.178302995 +0000 UTC m=+28.467959525" watchObservedRunningTime="2026-11-02 20:15:00.179052691 +0000 UTC m=+28.468709223" Nov 02 20:15:00.541634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:00.541585 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:00.795004 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:00.794933 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:00.998161 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:00.998123 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rphxh" Nov 02 20:15:01.255935 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:01.255897 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:01.392411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:01.392353 4971 patch_prober.go:29] interesting pod/image-registry-5d646d5849-btrvm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 20:15:01.392593 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:01.392475 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" podUID="9cb81e57-c7dc-4447-959f-d5496ffee976" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:15:01.513550 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:01.513481 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:02.758059 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:02.758033 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:03.017358 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:03.017259 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:03.086495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:03.086461 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b94zk" event={"ID":"1f6f9164-5e01-4400-a606-adf3c5016a61","Type":"ContainerStarted","Data":"26e876754d21dccae39f7021531770d052a95853d2fcffbf3cf2259f4d7e68cc"} Nov 02 20:15:03.087102 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:03.086780 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b94zk" Nov 02 20:15:03.092065 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:03.092039 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b94zk" Nov 02 20:15:03.197190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:03.197141 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-b94zk" podStartSLOduration=1.563426261 podStartE2EDuration="10.197127141s" totalImagesPullingTime="8.63370088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:03.195675797 +0000 UTC m=+31.485332319" watchObservedRunningTime="2026-11-02 20:15:03.197127141 +0000 UTC m=+31.486783652" Nov 02 20:15:03.343659 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:15:03.343528 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:15:03.382881 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:03.382838 4971 patch_prober.go:29] interesting pod/image-registry-6578786844-fgcrv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 20:15:03.383033 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:03.382895 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6578786844-fgcrv" podUID="02eded99-edf7-41a0-8866-549252be5fe0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:15:04.348947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:04.348900 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:04.601794 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:04.601766 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:04.860108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:04.860022 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:05.096871 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:05.096832 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799b587cd-csb9b" event={"ID":"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183","Type":"ContainerStarted","Data":"8a17d50f28cdba7448627fb7993c40171d6892c818aae95a16e68623f1b87479"} Nov 02 20:15:05.210281 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:05.210172 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-799b587cd-csb9b" podStartSLOduration=2.600043135 podStartE2EDuration="7.210158341s" totalImagesPullingTime="4.610115206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:05.209000566 +0000 UTC m=+33.498657098" watchObservedRunningTime="2026-11-02 20:15:05.210158341 +0000 UTC m=+33.499814868" Nov 02 20:15:05.885273 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:05.885238 4971 patch_prober.go:29] interesting pod/image-registry-5d646d5849-btrvm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 20:15:05.885736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:05.885301 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" podUID="9cb81e57-c7dc-4447-959f-d5496ffee976" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:15:05.885736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:05.885351 4971 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:15:05.885955 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:05.885885 4971 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"f514d4af192b5d56100c0ea6e339b5a63187275f0e34251c0173ba52fced8848"} pod="openshift-image-registry/image-registry-5d646d5849-btrvm" containerMessage="Container registry failed liveness probe, will be restarted" Nov 02 20:15:05.889208 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:05.889184 4971 patch_prober.go:29] interesting pod/image-registry-5d646d5849-btrvm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 20:15:05.889310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:05.889223 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" podUID="9cb81e57-c7dc-4447-959f-d5496ffee976" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:15:07.063980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:07.063945 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8lr5b" Nov 02 20:15:07.344685 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:07.344608 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:07.586058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:07.586025 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:08.285284 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:08.285252 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:08.554725 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:08.554641 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:08.799873 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:08.799838 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:09.037564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:09.037532 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:09.278704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:09.278673 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:15:09.278852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:09.278756 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:15:09.280244 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:09.280218 4971 patch_prober.go:29] interesting pod/console-799b587cd-csb9b container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.31:8443/health\": dial tcp 10.132.0.31:8443: connect: connection refused" start-of-body= Nov 02 20:15:09.280378 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:09.280269 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-console/console-799b587cd-csb9b" podUID="e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183" containerName="console" probeResult="failure" output="Get \"https://10.132.0.31:8443/health\": dial tcp 10.132.0.31:8443: connect: connection refused" Nov 02 20:15:10.887937 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:10.887901 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:11.630037 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:11.630001 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:11.881472 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:11.881395 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:12.132385 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.132318 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:12.273517 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.273473 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-869gg"] Nov 02 20:15:12.280140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.280117 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.374957 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.374926 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd2244bb-1e62-47be-9e7e-d80f39e2a525-metrics-client-ca\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.375066 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.374970 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cd2244bb-1e62-47be-9e7e-d80f39e2a525-node-exporter-accelerators-collector-config\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.375066 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.375028 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd2244bb-1e62-47be-9e7e-d80f39e2a525-sys\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.375202 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.375077 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cd2244bb-1e62-47be-9e7e-d80f39e2a525-node-exporter-textfile\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.375202 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.375186 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cd2244bb-1e62-47be-9e7e-d80f39e2a525-root\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.375289 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.375238 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkswv\" (UniqueName: \"kubernetes.io/projected/cd2244bb-1e62-47be-9e7e-d80f39e2a525-kube-api-access-gkswv\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.375337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.375322 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd2244bb-1e62-47be-9e7e-d80f39e2a525-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.375376 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.375354 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cd2244bb-1e62-47be-9e7e-d80f39e2a525-node-exporter-tls\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.375429 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.375381 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cd2244bb-1e62-47be-9e7e-d80f39e2a525-node-exporter-wtmp\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.379636 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.379617 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:12.437182 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.437124 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 20:15:12.437182 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.437125 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 20:15:12.437328 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.437269 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 20:15:12.437367 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.437335 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bvgts\"" Nov 02 20:15:12.437423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.437385 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 20:15:12.476788 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.476764 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd2244bb-1e62-47be-9e7e-d80f39e2a525-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.476889 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.476797 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cd2244bb-1e62-47be-9e7e-d80f39e2a525-node-exporter-tls\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.476889 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.476821 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cd2244bb-1e62-47be-9e7e-d80f39e2a525-node-exporter-wtmp\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.476889 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.476853 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd2244bb-1e62-47be-9e7e-d80f39e2a525-metrics-client-ca\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.476889 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.476879 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cd2244bb-1e62-47be-9e7e-d80f39e2a525-node-exporter-accelerators-collector-config\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.477101 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.476908 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd2244bb-1e62-47be-9e7e-d80f39e2a525-sys\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.477101 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.476957 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cd2244bb-1e62-47be-9e7e-d80f39e2a525-node-exporter-textfile\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.477101 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.476978 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cd2244bb-1e62-47be-9e7e-d80f39e2a525-node-exporter-wtmp\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.477101 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.477003 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cd2244bb-1e62-47be-9e7e-d80f39e2a525-root\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.477101 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.477025 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkswv\" (UniqueName: \"kubernetes.io/projected/cd2244bb-1e62-47be-9e7e-d80f39e2a525-kube-api-access-gkswv\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.477101 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.477032 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd2244bb-1e62-47be-9e7e-d80f39e2a525-sys\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.477374 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.477108 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cd2244bb-1e62-47be-9e7e-d80f39e2a525-root\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.477374 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.477319 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cd2244bb-1e62-47be-9e7e-d80f39e2a525-node-exporter-textfile\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.477513 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.477498 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd2244bb-1e62-47be-9e7e-d80f39e2a525-metrics-client-ca\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.477763 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.477740 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cd2244bb-1e62-47be-9e7e-d80f39e2a525-node-exporter-accelerators-collector-config\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.479285 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.479263 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd2244bb-1e62-47be-9e7e-d80f39e2a525-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.479555 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.479508 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cd2244bb-1e62-47be-9e7e-d80f39e2a525-node-exporter-tls\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.534983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.534961 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkswv\" (UniqueName: \"kubernetes.io/projected/cd2244bb-1e62-47be-9e7e-d80f39e2a525-kube-api-access-gkswv\") pod \"node-exporter-869gg\" (UID: \"cd2244bb-1e62-47be-9e7e-d80f39e2a525\") " pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.599190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.599160 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-869gg" Nov 02 20:15:12.611385 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:15:12.611358 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2244bb_1e62_47be_9e7e_d80f39e2a525.slice/crio-7bf1e3a91f89a39bf5c35f024db1174c74bc1fe658363d21290aa941f1bbbb98 WatchSource:0}: Error finding container 7bf1e3a91f89a39bf5c35f024db1174c74bc1fe658363d21290aa941f1bbbb98: Status 404 returned error can't find the container with id 7bf1e3a91f89a39bf5c35f024db1174c74bc1fe658363d21290aa941f1bbbb98 Nov 02 20:15:12.622969 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.622941 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:12.870466 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:12.870438 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:13.113106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:13.113061 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:13.133153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:13.133072 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-869gg" event={"ID":"cd2244bb-1e62-47be-9e7e-d80f39e2a525","Type":"ContainerStarted","Data":"7bf1e3a91f89a39bf5c35f024db1174c74bc1fe658363d21290aa941f1bbbb98"} Nov 02 20:15:13.381268 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:13.381234 4971 patch_prober.go:29] interesting pod/image-registry-6578786844-fgcrv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 20:15:13.381409 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:13.381282 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6578786844-fgcrv" podUID="02eded99-edf7-41a0-8866-549252be5fe0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:15:13.638943 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:13.638911 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:13.734657 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:13.734628 4971 patch_prober.go:29] interesting pod/image-registry-dc7dd847c-sl2nq container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 20:15:13.734882 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:13.734679 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" podUID="360a57b3-cbd3-4620-a9b0-222d457a085c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:15:13.889380 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:13.889291 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:14.502780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:14.502743 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:15.143656 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:15.143624 4971 generic.go:356] "Generic (PLEG): container finished" podID="cd2244bb-1e62-47be-9e7e-d80f39e2a525" containerID="1571816a6abe74b76a0b2c3bc443f011cbf60df0dd088fe9eb791b131859d105" exitCode=0 Nov 02 20:15:15.143811 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:15.143680 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-869gg" event={"ID":"cd2244bb-1e62-47be-9e7e-d80f39e2a525","Type":"ContainerDied","Data":"1571816a6abe74b76a0b2c3bc443f011cbf60df0dd088fe9eb791b131859d105"} Nov 02 20:15:15.145896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:15.145824 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:15.393474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:15.393443 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:15.889920 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:15.889890 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:15:16.052864 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:16.052839 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-dc7dd847c-sl2nq" Nov 02 20:15:16.150361 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:16.150286 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-869gg" event={"ID":"cd2244bb-1e62-47be-9e7e-d80f39e2a525","Type":"ContainerStarted","Data":"e2f75cf75c5db4c24cd4ce6daa633848f8780261e3649b919e0fd3613de107a9"} Nov 02 20:15:16.150361 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:16.150329 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-869gg" event={"ID":"cd2244bb-1e62-47be-9e7e-d80f39e2a525","Type":"ContainerStarted","Data":"fdb77380e4e1a50bed7541e13de5e0b95bff8fd8529a16fd14ca26c8b777ee94"} Nov 02 20:15:16.196591 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:16.196555 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d646d5849-btrvm"] Nov 02 20:15:16.418943 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:16.418860 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:16.670596 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:16.670517 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:16.915695 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:16.915662 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:17.166346 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:17.166301 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:17.412222 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:17.412174 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:18.046258 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.046204 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-869gg" podStartSLOduration=4.216795582 podStartE2EDuration="6.046182878s" totalImagesPullingTime="1.752766318s" totalInitContainerRuntime="76.620978ms" isStatefulPod=true podCreationTimestamp="2026-11-02 20:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:16.283652766 +0000 UTC m=+44.573309288" watchObservedRunningTime="2026-11-02 20:15:18.046182878 +0000 UTC m=+46.335839405" Nov 02 20:15:18.046711 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.046408 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-799b587cd-csb9b"] Nov 02 20:15:18.397433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.397397 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6578786844-fgcrv" podUID="02eded99-edf7-41a0-8866-549252be5fe0" containerName="registry" containerID="cri-o://d2a3afe03bb144cd81b315511993bbecb17e5e3b4e914cbdbb2c2af5a0621b1b" gracePeriod=30 Nov 02 20:15:18.646034 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.646010 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:15:18.730655 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.730595 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-trusted-ca\") pod \"02eded99-edf7-41a0-8866-549252be5fe0\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " Nov 02 20:15:18.730655 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.730641 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-image-registry-private-configuration\") pod \"02eded99-edf7-41a0-8866-549252be5fe0\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " Nov 02 20:15:18.730854 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.730693 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-registry-certificates\") pod \"02eded99-edf7-41a0-8866-549252be5fe0\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " Nov 02 20:15:18.730854 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.730720 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-kube-api-access-kmsvc\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-kube-api-access-kmsvc\") pod \"02eded99-edf7-41a0-8866-549252be5fe0\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " Nov 02 20:15:18.730854 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.730752 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-bound-sa-token\") pod \"02eded99-edf7-41a0-8866-549252be5fe0\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " Nov 02 20:15:18.730854 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.730785 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02eded99-edf7-41a0-8866-549252be5fe0-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02eded99-edf7-41a0-8866-549252be5fe0-ca-trust-extracted\") pod \"02eded99-edf7-41a0-8866-549252be5fe0\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " Nov 02 20:15:18.730854 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.730846 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls\") pod \"02eded99-edf7-41a0-8866-549252be5fe0\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " Nov 02 20:15:18.731139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.730878 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-installation-pull-secrets\") pod \"02eded99-edf7-41a0-8866-549252be5fe0\" (UID: \"02eded99-edf7-41a0-8866-549252be5fe0\") " Nov 02 20:15:18.731139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.731056 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-trusted-ca" pod "02eded99-edf7-41a0-8866-549252be5fe0" (UID: "02eded99-edf7-41a0-8866-549252be5fe0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:18.731244 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.731148 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-registry-certificates" pod "02eded99-edf7-41a0-8866-549252be5fe0" (UID: "02eded99-edf7-41a0-8866-549252be5fe0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:18.731244 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.731201 4971 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-trusted-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.734045 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.734022 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-image-registry-private-configuration" pod "02eded99-edf7-41a0-8866-549252be5fe0" (UID: "02eded99-edf7-41a0-8866-549252be5fe0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:18.734185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.734021 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls" pod "02eded99-edf7-41a0-8866-549252be5fe0" (UID: "02eded99-edf7-41a0-8866-549252be5fe0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:18.734244 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.734189 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-kube-api-access-kmsvc" pod "02eded99-edf7-41a0-8866-549252be5fe0" (UID: "02eded99-edf7-41a0-8866-549252be5fe0"). InnerVolumeSpecName "kube-api-access-kmsvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:18.734524 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.734464 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-installation-pull-secrets" pod "02eded99-edf7-41a0-8866-549252be5fe0" (UID: "02eded99-edf7-41a0-8866-549252be5fe0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:18.734524 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.734491 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-bound-sa-token" pod "02eded99-edf7-41a0-8866-549252be5fe0" (UID: "02eded99-edf7-41a0-8866-549252be5fe0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:18.739029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.739005 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02eded99-edf7-41a0-8866-549252be5fe0-ca-trust-extracted" pod "02eded99-edf7-41a0-8866-549252be5fe0" (UID: "02eded99-edf7-41a0-8866-549252be5fe0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:18.832422 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.832388 4971 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-registry-tls\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.832422 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.832411 4971 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-installation-pull-secrets\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.832422 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.832422 4971 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02eded99-edf7-41a0-8866-549252be5fe0-image-registry-private-configuration\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.832722 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.832432 4971 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02eded99-edf7-41a0-8866-549252be5fe0-registry-certificates\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.832722 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.832440 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmsvc\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-kube-api-access-kmsvc\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.832722 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.832449 4971 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02eded99-edf7-41a0-8866-549252be5fe0-bound-sa-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.832722 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:18.832458 4971 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02eded99-edf7-41a0-8866-549252be5fe0-ca-trust-extracted\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:19.165022 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:19.164985 4971 generic.go:356] "Generic (PLEG): container finished" podID="02eded99-edf7-41a0-8866-549252be5fe0" containerID="d2a3afe03bb144cd81b315511993bbecb17e5e3b4e914cbdbb2c2af5a0621b1b" exitCode=0 Nov 02 20:15:19.165495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:19.165035 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6578786844-fgcrv" Nov 02 20:15:19.165495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:19.165076 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6578786844-fgcrv" event={"ID":"02eded99-edf7-41a0-8866-549252be5fe0","Type":"ContainerDied","Data":"d2a3afe03bb144cd81b315511993bbecb17e5e3b4e914cbdbb2c2af5a0621b1b"} Nov 02 20:15:19.165495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:19.165158 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6578786844-fgcrv" event={"ID":"02eded99-edf7-41a0-8866-549252be5fe0","Type":"ContainerDied","Data":"9f32bbd14e766f8b66dcd4613e4ebd480a8b133714cb69d6cf69ff30a8af3de5"} Nov 02 20:15:19.165495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:19.165184 4971 scope.go:122] "RemoveContainer" containerID="d2a3afe03bb144cd81b315511993bbecb17e5e3b4e914cbdbb2c2af5a0621b1b" Nov 02 20:15:19.174705 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:19.174688 4971 scope.go:122] "RemoveContainer" containerID="d2a3afe03bb144cd81b315511993bbecb17e5e3b4e914cbdbb2c2af5a0621b1b" Nov 02 20:15:19.174978 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:15:19.174959 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a3afe03bb144cd81b315511993bbecb17e5e3b4e914cbdbb2c2af5a0621b1b\": container with ID starting with d2a3afe03bb144cd81b315511993bbecb17e5e3b4e914cbdbb2c2af5a0621b1b not found: ID does not exist" containerID="d2a3afe03bb144cd81b315511993bbecb17e5e3b4e914cbdbb2c2af5a0621b1b" Nov 02 20:15:19.175025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:19.174987 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a3afe03bb144cd81b315511993bbecb17e5e3b4e914cbdbb2c2af5a0621b1b"} err="failed to get container status \"d2a3afe03bb144cd81b315511993bbecb17e5e3b4e914cbdbb2c2af5a0621b1b\": rpc error: code = NotFound desc = could not find container \"d2a3afe03bb144cd81b315511993bbecb17e5e3b4e914cbdbb2c2af5a0621b1b\": container with ID starting with d2a3afe03bb144cd81b315511993bbecb17e5e3b4e914cbdbb2c2af5a0621b1b not found: ID does not exist" Nov 02 20:15:19.320147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:19.320118 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6578786844-fgcrv"] Nov 02 20:15:19.366779 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:19.366754 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6578786844-fgcrv"] Nov 02 20:15:19.864988 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:19.864958 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02eded99-edf7-41a0-8866-549252be5fe0" path="/var/lib/kubelet/pods/02eded99-edf7-41a0-8866-549252be5fe0/volumes" Nov 02 20:15:21.240928 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:21.240890 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:21.493265 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:21.493195 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:21.749701 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:21.749633 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:22.015657 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:22.015566 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:23.665231 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:23.665200 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:23.909982 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:23.909946 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:25.434214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:25.434175 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:30.908825 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:30.908782 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" podUID="9cb81e57-c7dc-4447-959f-d5496ffee976" containerName="registry" containerID="cri-o://f514d4af192b5d56100c0ea6e339b5a63187275f0e34251c0173ba52fced8848" gracePeriod=30 Nov 02 20:15:31.220926 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:31.220847 4971 generic.go:356] "Generic (PLEG): container finished" podID="9cb81e57-c7dc-4447-959f-d5496ffee976" containerID="f514d4af192b5d56100c0ea6e339b5a63187275f0e34251c0173ba52fced8848" exitCode=0 Nov 02 20:15:31.221154 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:31.220930 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" event={"ID":"9cb81e57-c7dc-4447-959f-d5496ffee976","Type":"ContainerDied","Data":"f514d4af192b5d56100c0ea6e339b5a63187275f0e34251c0173ba52fced8848"} Nov 02 20:15:31.221154 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:31.220980 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" event={"ID":"9cb81e57-c7dc-4447-959f-d5496ffee976","Type":"ContainerStarted","Data":"8e02001a7b1d44002c5aef08bf8337429536d56b706360f21ee4d5375d2836ba"} Nov 02 20:15:31.221154 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:31.221107 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:15:31.751687 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:31.751652 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:32.007983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:32.007900 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:33.339843 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:15:33.339806 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:15:37.252834 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:37.252784 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-wfwqz" event={"ID":"69b4285c-bc77-47f9-ae3e-33210d640283","Type":"ContainerStarted","Data":"c9a7e0393935ab3595a8df1f82b3a5a14cf649e57afda37dfc69dbc817fd369b"} Nov 02 20:15:37.253390 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:37.252987 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-wfwqz" Nov 02 20:15:37.276576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:37.276544 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-wfwqz" Nov 02 20:15:37.413103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:37.413014 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-wfwqz" podStartSLOduration=2.131578904 podStartE2EDuration="44.41299988s" totalImagesPullingTime="42.281420976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:37.411779914 +0000 UTC m=+65.701436441" watchObservedRunningTime="2026-11-02 20:15:37.41299988 +0000 UTC m=+65.702656407" Nov 02 20:15:38.671800 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:38.671762 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:38.916044 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:38.916001 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:39.164988 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:39.164954 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:39.420959 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:39.420881 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:43.066890 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.066817 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-799b587cd-csb9b" podUID="e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183" containerName="console" containerID="cri-o://8a17d50f28cdba7448627fb7993c40171d6892c818aae95a16e68623f1b87479" gracePeriod=15 Nov 02 20:15:43.282825 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.282752 4971 generic.go:356] "Generic (PLEG): container finished" podID="e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183" containerID="8a17d50f28cdba7448627fb7993c40171d6892c818aae95a16e68623f1b87479" exitCode=0 Nov 02 20:15:43.282991 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.282831 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799b587cd-csb9b" event={"ID":"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183","Type":"ContainerDied","Data":"8a17d50f28cdba7448627fb7993c40171d6892c818aae95a16e68623f1b87479"} Nov 02 20:15:43.384896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.384868 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:15:43.546185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.546144 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-service-ca\") pod \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " Nov 02 20:15:43.546564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.546194 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-config\" (UniqueName: \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-config\") pod \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " Nov 02 20:15:43.546564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.546222 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-oauth-serving-cert\") pod \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " Nov 02 20:15:43.546564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.546286 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-kube-api-access-8lcsf\" (UniqueName: \"kubernetes.io/projected/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-kube-api-access-8lcsf\") pod \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " Nov 02 20:15:43.546564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.546325 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-oauth-config\") pod \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " Nov 02 20:15:43.546779 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.546585 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-tmp\") pod \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " Nov 02 20:15:43.546779 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.546641 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-service-ca" pod "e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183" (UID: "e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:43.546779 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.546651 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-serving-cert\") pod \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\" (UID: \"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183\") " Nov 02 20:15:43.547849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.546988 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-tmp" pod "e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183" (UID: "e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:43.547849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.547168 4971 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-tmp\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.547849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.547192 4971 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-service-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.547849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.547358 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-oauth-serving-cert" pod "e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183" (UID: "e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:43.547849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.547790 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-config" pod "e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183" (UID: "e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:43.549534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.549506 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-kube-api-access-8lcsf" pod "e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183" (UID: "e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183"). InnerVolumeSpecName "kube-api-access-8lcsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:43.549534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.549526 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-oauth-config" pod "e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183" (UID: "e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:43.549909 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.549889 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-serving-cert" pod "e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183" (UID: "e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:43.647626 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.647595 4971 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-oauth-config\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.647626 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.647624 4971 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-serving-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.647849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.647636 4971 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-console-config\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.647849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.647646 4971 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-oauth-serving-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:43.647849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:43.647655 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lcsf\" (UniqueName: \"kubernetes.io/projected/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183-kube-api-access-8lcsf\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.289459 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:44.289423 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799b587cd-csb9b" event={"ID":"e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183","Type":"ContainerDied","Data":"5be09a529ddfb8e3e5131e840835f24f2937b0bad28eacb184dcf1a20f6d68db"} Nov 02 20:15:44.289459 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:44.289457 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799b587cd-csb9b" Nov 02 20:15:44.290015 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:44.289496 4971 scope.go:122] "RemoveContainer" containerID="8a17d50f28cdba7448627fb7993c40171d6892c818aae95a16e68623f1b87479" Nov 02 20:15:44.524448 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:44.524412 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-799b587cd-csb9b"] Nov 02 20:15:44.528249 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:44.528221 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-799b587cd-csb9b"] Nov 02 20:15:44.838399 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:44.838362 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:45.082055 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:45.082016 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:45.864335 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:45.864302 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183" path="/var/lib/kubelet/pods/e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183/volumes" Nov 02 20:15:50.315000 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:50.314926 4971 generic.go:356] "Generic (PLEG): container finished" podID="d23cf40e-53e9-406e-92f8-ce628b1baa68" containerID="ef2a96e5640f307ff6a9eda12e05fe8ca8394e911cfa974de5d6f2217306c907" exitCode=0 Nov 02 20:15:50.315457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:50.315000 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" event={"ID":"d23cf40e-53e9-406e-92f8-ce628b1baa68","Type":"ContainerDied","Data":"ef2a96e5640f307ff6a9eda12e05fe8ca8394e911cfa974de5d6f2217306c907"} Nov 02 20:15:50.315457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:50.315297 4971 scope.go:122] "RemoveContainer" containerID="ef2a96e5640f307ff6a9eda12e05fe8ca8394e911cfa974de5d6f2217306c907" Nov 02 20:15:51.226156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:51.226129 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:15:51.320927 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:51.320902 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-pv8n6" event={"ID":"d23cf40e-53e9-406e-92f8-ce628b1baa68","Type":"ContainerStarted","Data":"0a1a216d47efbd7fade2bcc83c36003d707f8d526e14519c3755dc40c90cf222"} Nov 02 20:15:53.330820 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:53.330784 4971 generic.go:356] "Generic (PLEG): container finished" podID="f65dc226-70a9-4fcf-ae3b-28559c2ff235" containerID="154c208bc990652c55848959ffa45522d8d88c087f9e091c5dec3a7bca8587f8" exitCode=0 Nov 02 20:15:53.331324 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:53.330865 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-v9wrk" event={"ID":"f65dc226-70a9-4fcf-ae3b-28559c2ff235","Type":"ContainerDied","Data":"154c208bc990652c55848959ffa45522d8d88c087f9e091c5dec3a7bca8587f8"} Nov 02 20:15:53.331324 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:53.331261 4971 scope.go:122] "RemoveContainer" containerID="154c208bc990652c55848959ffa45522d8d88c087f9e091c5dec3a7bca8587f8" Nov 02 20:15:53.668435 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:53.668389 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:53.919025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:53.918948 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:54.337271 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.337197 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-v9wrk" event={"ID":"f65dc226-70a9-4fcf-ae3b-28559c2ff235","Type":"ContainerStarted","Data":"9915eca01eb5774ca7a11e544ab0b45ae2b14972c7b3c33cd4649d965585f65d"} Nov 02 20:15:54.375797 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.375769 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hxckx-j8tzv"] Nov 02 20:15:54.375953 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.375935 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-hxckx-j8tzv" podUID="4d75d87e-395e-4668-8b7a-42bdd79d08d9" containerName="insights-gathering" containerID="cri-o://26850d4a4881a050a1e914f96360fcec2e0aaeab155729e24d2844f83e6d91fe" gracePeriod=30 Nov 02 20:15:54.684307 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.684283 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:15:54.833293 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.833262 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-service-ca-bundle\") pod \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " Nov 02 20:15:54.833432 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.833310 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d75d87e-395e-4668-8b7a-42bdd79d08d9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4d75d87e-395e-4668-8b7a-42bdd79d08d9-archives-path\") pod \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " Nov 02 20:15:54.833432 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.833385 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d75d87e-395e-4668-8b7a-42bdd79d08d9-kube-api-access-ltqcp\" (UniqueName: \"kubernetes.io/projected/4d75d87e-395e-4668-8b7a-42bdd79d08d9-kube-api-access-ltqcp\") pod \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " Nov 02 20:15:54.833432 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.833401 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-trusted-ca-bundle\") pod \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\" (UID: \"4d75d87e-395e-4668-8b7a-42bdd79d08d9\") " Nov 02 20:15:54.833723 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.833697 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d75d87e-395e-4668-8b7a-42bdd79d08d9-archives-path" pod "4d75d87e-395e-4668-8b7a-42bdd79d08d9" (UID: "4d75d87e-395e-4668-8b7a-42bdd79d08d9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:54.833723 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.833708 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-service-ca-bundle" pod "4d75d87e-395e-4668-8b7a-42bdd79d08d9" (UID: "4d75d87e-395e-4668-8b7a-42bdd79d08d9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:54.833896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.833863 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-trusted-ca-bundle" pod "4d75d87e-395e-4668-8b7a-42bdd79d08d9" (UID: "4d75d87e-395e-4668-8b7a-42bdd79d08d9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:54.835760 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.835738 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d75d87e-395e-4668-8b7a-42bdd79d08d9-kube-api-access-ltqcp" pod "4d75d87e-395e-4668-8b7a-42bdd79d08d9" (UID: "4d75d87e-395e-4668-8b7a-42bdd79d08d9"). InnerVolumeSpecName "kube-api-access-ltqcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:54.909563 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.909504 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:54.934701 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.934679 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltqcp\" (UniqueName: \"kubernetes.io/projected/4d75d87e-395e-4668-8b7a-42bdd79d08d9-kube-api-access-ltqcp\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:54.934701 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.934701 4971 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-trusted-ca-bundle\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:54.934848 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.934711 4971 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d75d87e-395e-4668-8b7a-42bdd79d08d9-service-ca-bundle\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:54.934848 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:54.934720 4971 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4d75d87e-395e-4668-8b7a-42bdd79d08d9-archives-path\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:55.177975 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:55.177909 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:15:55.342150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:55.342121 4971 generic.go:356] "Generic (PLEG): container finished" podID="4d75d87e-395e-4668-8b7a-42bdd79d08d9" containerID="26850d4a4881a050a1e914f96360fcec2e0aaeab155729e24d2844f83e6d91fe" exitCode=2 Nov 02 20:15:55.342507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:55.342169 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hxckx-j8tzv" Nov 02 20:15:55.342507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:55.342205 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hxckx-j8tzv" event={"ID":"4d75d87e-395e-4668-8b7a-42bdd79d08d9","Type":"ContainerDied","Data":"26850d4a4881a050a1e914f96360fcec2e0aaeab155729e24d2844f83e6d91fe"} Nov 02 20:15:55.342507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:55.342267 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hxckx-j8tzv" event={"ID":"4d75d87e-395e-4668-8b7a-42bdd79d08d9","Type":"ContainerDied","Data":"d6151eb6e612794d3cdae41786b68fae9972d326cb1b3e497e02d5d72f9bf66f"} Nov 02 20:15:55.342507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:55.342296 4971 scope.go:122] "RemoveContainer" containerID="26850d4a4881a050a1e914f96360fcec2e0aaeab155729e24d2844f83e6d91fe" Nov 02 20:15:55.354281 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:55.354232 4971 scope.go:122] "RemoveContainer" containerID="26850d4a4881a050a1e914f96360fcec2e0aaeab155729e24d2844f83e6d91fe" Nov 02 20:15:55.354611 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:15:55.354587 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26850d4a4881a050a1e914f96360fcec2e0aaeab155729e24d2844f83e6d91fe\": container with ID starting with 26850d4a4881a050a1e914f96360fcec2e0aaeab155729e24d2844f83e6d91fe not found: ID does not exist" containerID="26850d4a4881a050a1e914f96360fcec2e0aaeab155729e24d2844f83e6d91fe" Nov 02 20:15:55.354693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:55.354620 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26850d4a4881a050a1e914f96360fcec2e0aaeab155729e24d2844f83e6d91fe"} err="failed to get container status \"26850d4a4881a050a1e914f96360fcec2e0aaeab155729e24d2844f83e6d91fe\": rpc error: code = NotFound desc = could not find container \"26850d4a4881a050a1e914f96360fcec2e0aaeab155729e24d2844f83e6d91fe\": container with ID starting with 26850d4a4881a050a1e914f96360fcec2e0aaeab155729e24d2844f83e6d91fe not found: ID does not exist" Nov 02 20:15:55.496041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:55.495986 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hxckx-j8tzv"] Nov 02 20:15:55.543136 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:55.543112 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-hxckx-j8tzv"] Nov 02 20:15:55.864574 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:55.864544 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d75d87e-395e-4668-8b7a-42bdd79d08d9" path="/var/lib/kubelet/pods/4d75d87e-395e-4668-8b7a-42bdd79d08d9/volumes" Nov 02 20:15:56.243953 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.243881 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" podUID="9cb81e57-c7dc-4447-959f-d5496ffee976" containerName="registry" containerID="cri-o://8e02001a7b1d44002c5aef08bf8337429536d56b706360f21ee4d5375d2836ba" gracePeriod=30 Nov 02 20:15:56.499495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.499437 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:15:56.646604 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.646572 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-trusted-ca\") pod \"9cb81e57-c7dc-4447-959f-d5496ffee976\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " Nov 02 20:15:56.646724 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.646635 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-kube-api-access-xpqp8\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-kube-api-access-xpqp8\") pod \"9cb81e57-c7dc-4447-959f-d5496ffee976\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " Nov 02 20:15:56.646724 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.646662 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-image-registry-private-configuration\") pod \"9cb81e57-c7dc-4447-959f-d5496ffee976\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " Nov 02 20:15:56.646724 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.646683 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-bound-sa-token\") pod \"9cb81e57-c7dc-4447-959f-d5496ffee976\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " Nov 02 20:15:56.646724 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.646703 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls\") pod \"9cb81e57-c7dc-4447-959f-d5496ffee976\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " Nov 02 20:15:56.646948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.646726 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb81e57-c7dc-4447-959f-d5496ffee976-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cb81e57-c7dc-4447-959f-d5496ffee976-ca-trust-extracted\") pod \"9cb81e57-c7dc-4447-959f-d5496ffee976\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " Nov 02 20:15:56.646948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.646761 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-installation-pull-secrets\") pod \"9cb81e57-c7dc-4447-959f-d5496ffee976\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " Nov 02 20:15:56.646948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.646936 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-certificates\") pod \"9cb81e57-c7dc-4447-959f-d5496ffee976\" (UID: \"9cb81e57-c7dc-4447-959f-d5496ffee976\") " Nov 02 20:15:56.647204 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.647055 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-trusted-ca" pod "9cb81e57-c7dc-4447-959f-d5496ffee976" (UID: "9cb81e57-c7dc-4447-959f-d5496ffee976"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:56.647428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.647390 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-certificates" pod "9cb81e57-c7dc-4447-959f-d5496ffee976" (UID: "9cb81e57-c7dc-4447-959f-d5496ffee976"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:56.649458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.649419 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-installation-pull-secrets" pod "9cb81e57-c7dc-4447-959f-d5496ffee976" (UID: "9cb81e57-c7dc-4447-959f-d5496ffee976"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:56.649562 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.649471 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls" pod "9cb81e57-c7dc-4447-959f-d5496ffee976" (UID: "9cb81e57-c7dc-4447-959f-d5496ffee976"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:56.649672 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.649632 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-kube-api-access-xpqp8" pod "9cb81e57-c7dc-4447-959f-d5496ffee976" (UID: "9cb81e57-c7dc-4447-959f-d5496ffee976"). InnerVolumeSpecName "kube-api-access-xpqp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:56.649814 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.649793 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-bound-sa-token" pod "9cb81e57-c7dc-4447-959f-d5496ffee976" (UID: "9cb81e57-c7dc-4447-959f-d5496ffee976"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:56.649814 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.649803 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-image-registry-private-configuration" pod "9cb81e57-c7dc-4447-959f-d5496ffee976" (UID: "9cb81e57-c7dc-4447-959f-d5496ffee976"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:56.654044 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.654019 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb81e57-c7dc-4447-959f-d5496ffee976-ca-trust-extracted" pod "9cb81e57-c7dc-4447-959f-d5496ffee976" (UID: "9cb81e57-c7dc-4447-959f-d5496ffee976"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:56.748336 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.748305 4971 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-installation-pull-secrets\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:56.748336 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.748339 4971 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-certificates\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:56.748503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.748355 4971 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9cb81e57-c7dc-4447-959f-d5496ffee976-trusted-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:56.748503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.748369 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpqp8\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-kube-api-access-xpqp8\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:56.748503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.748385 4971 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9cb81e57-c7dc-4447-959f-d5496ffee976-image-registry-private-configuration\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:56.748503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.748399 4971 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-bound-sa-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:56.748503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.748413 4971 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9cb81e57-c7dc-4447-959f-d5496ffee976-registry-tls\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:56.748503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:56.748427 4971 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9cb81e57-c7dc-4447-959f-d5496ffee976-ca-trust-extracted\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:57.352753 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:57.352721 4971 generic.go:356] "Generic (PLEG): container finished" podID="9cb81e57-c7dc-4447-959f-d5496ffee976" containerID="8e02001a7b1d44002c5aef08bf8337429536d56b706360f21ee4d5375d2836ba" exitCode=0 Nov 02 20:15:57.352945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:57.352772 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" Nov 02 20:15:57.352945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:57.352782 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" event={"ID":"9cb81e57-c7dc-4447-959f-d5496ffee976","Type":"ContainerDied","Data":"8e02001a7b1d44002c5aef08bf8337429536d56b706360f21ee4d5375d2836ba"} Nov 02 20:15:57.352945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:57.352823 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d646d5849-btrvm" event={"ID":"9cb81e57-c7dc-4447-959f-d5496ffee976","Type":"ContainerDied","Data":"05548d62c130fd64f54c8c1b9903be3863527c077ee4ca5f98becadcb2948dc9"} Nov 02 20:15:57.352945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:57.352843 4971 scope.go:122] "RemoveContainer" containerID="8e02001a7b1d44002c5aef08bf8337429536d56b706360f21ee4d5375d2836ba" Nov 02 20:15:57.361897 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:57.361876 4971 scope.go:122] "RemoveContainer" containerID="f514d4af192b5d56100c0ea6e339b5a63187275f0e34251c0173ba52fced8848" Nov 02 20:15:57.369232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:57.369217 4971 scope.go:122] "RemoveContainer" containerID="8e02001a7b1d44002c5aef08bf8337429536d56b706360f21ee4d5375d2836ba" Nov 02 20:15:57.369483 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:15:57.369464 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e02001a7b1d44002c5aef08bf8337429536d56b706360f21ee4d5375d2836ba\": container with ID starting with 8e02001a7b1d44002c5aef08bf8337429536d56b706360f21ee4d5375d2836ba not found: ID does not exist" containerID="8e02001a7b1d44002c5aef08bf8337429536d56b706360f21ee4d5375d2836ba" Nov 02 20:15:57.369536 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:57.369492 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e02001a7b1d44002c5aef08bf8337429536d56b706360f21ee4d5375d2836ba"} err="failed to get container status \"8e02001a7b1d44002c5aef08bf8337429536d56b706360f21ee4d5375d2836ba\": rpc error: code = NotFound desc = could not find container \"8e02001a7b1d44002c5aef08bf8337429536d56b706360f21ee4d5375d2836ba\": container with ID starting with 8e02001a7b1d44002c5aef08bf8337429536d56b706360f21ee4d5375d2836ba not found: ID does not exist" Nov 02 20:15:57.369536 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:57.369508 4971 scope.go:122] "RemoveContainer" containerID="f514d4af192b5d56100c0ea6e339b5a63187275f0e34251c0173ba52fced8848" Nov 02 20:15:57.369730 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:15:57.369714 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f514d4af192b5d56100c0ea6e339b5a63187275f0e34251c0173ba52fced8848\": container with ID starting with f514d4af192b5d56100c0ea6e339b5a63187275f0e34251c0173ba52fced8848 not found: ID does not exist" containerID="f514d4af192b5d56100c0ea6e339b5a63187275f0e34251c0173ba52fced8848" Nov 02 20:15:57.369766 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:57.369735 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f514d4af192b5d56100c0ea6e339b5a63187275f0e34251c0173ba52fced8848"} err="failed to get container status \"f514d4af192b5d56100c0ea6e339b5a63187275f0e34251c0173ba52fced8848\": rpc error: code = NotFound desc = could not find container \"f514d4af192b5d56100c0ea6e339b5a63187275f0e34251c0173ba52fced8848\": container with ID starting with f514d4af192b5d56100c0ea6e339b5a63187275f0e34251c0173ba52fced8848 not found: ID does not exist" Nov 02 20:15:57.554858 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:57.554835 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d646d5849-btrvm"] Nov 02 20:15:57.601838 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:57.601812 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d646d5849-btrvm"] Nov 02 20:15:57.864244 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:15:57.864217 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb81e57-c7dc-4447-959f-d5496ffee976" path="/var/lib/kubelet/pods/9cb81e57-c7dc-4447-959f-d5496ffee976/volumes" Nov 02 20:16:03.335219 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:16:03.335184 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:16:03.379673 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:16:03.379646 4971 generic.go:356] "Generic (PLEG): container finished" podID="896fb858-68d1-462c-8a00-2b2b6b5a01af" containerID="076f669bd772da8292e2d7f35ae135673036b5faa09532ac3b60b45fcb2ba3b7" exitCode=0 Nov 02 20:16:03.379793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:16:03.379719 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" event={"ID":"896fb858-68d1-462c-8a00-2b2b6b5a01af","Type":"ContainerDied","Data":"076f669bd772da8292e2d7f35ae135673036b5faa09532ac3b60b45fcb2ba3b7"} Nov 02 20:16:03.380038 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:16:03.380023 4971 scope.go:122] "RemoveContainer" containerID="076f669bd772da8292e2d7f35ae135673036b5faa09532ac3b60b45fcb2ba3b7" Nov 02 20:16:04.008231 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:16:04.008200 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:04.258098 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:16:04.258056 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:04.385253 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:16:04.385222 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6qvvf" event={"ID":"896fb858-68d1-462c-8a00-2b2b6b5a01af","Type":"ContainerStarted","Data":"f323dec28fccf4a0a95c18d6d3a60289aa1f57e0fa8681c7f6864fce6f9efacd"} Nov 02 20:16:08.681257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:16:08.681227 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:08.925758 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:16:08.925719 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:15.369982 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:16:15.369942 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:15.621299 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:16:15.621225 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:15.871014 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:16:15.870973 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:22.185141 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:16:22.185111 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:22.477004 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:16:22.476932 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:16:33.337574 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:16:33.337531 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:17:01.757593 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:17:01.757501 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:02.005732 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:17:02.005689 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:03.336822 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:17:03.336791 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:17:06.612894 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:17:06.612857 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:14.461041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:17:14.460996 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:14.728877 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:17:14.728801 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:22.560331 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:17:22.560291 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:23.704707 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:17:23.704677 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:23.945329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:17:23.945295 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:33.337229 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:17:33.337191 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:17:48.231734 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:17:48.231695 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:48.483982 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:17:48.483877 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:49.493052 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:17:49.493017 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:17:49.736906 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:17:49.736857 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:03.337046 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:18:03.337009 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:18:22.873571 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:18:22.873496 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:18:33.336959 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:18:33.336919 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:19:03.336168 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:19:03.336130 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:19:09.844988 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:09.844957 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:11.730568 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:11.730527 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:13.534861 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:13.534832 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:23.215021 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:23.214982 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:23.460900 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:23.460871 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:26.032960 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:26.032928 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:26.316762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:26.316669 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:27.472016 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:27.471986 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:27.725253 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:27.725175 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:28.788215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:28.788183 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:31.877178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:31.877145 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:19:31.878196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:31.878175 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:19:33.336798 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:19:33.336761 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:19:34.839888 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:34.839842 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:35.114926 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:35.114895 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:36.224744 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.224713 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws"] Nov 02 20:19:36.225228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.225038 4971 cpu_manager.go:414] "containerMap: removing container" podUID="9cb81e57-c7dc-4447-959f-d5496ffee976" containerName="registry" Nov 02 20:19:36.225228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.225049 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb81e57-c7dc-4447-959f-d5496ffee976" containerName="registry" Nov 02 20:19:36.225228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.225057 4971 cpu_manager.go:414] "containerMap: removing container" podUID="e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183" containerName="console" Nov 02 20:19:36.225228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.225062 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183" containerName="console" Nov 02 20:19:36.225228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.225073 4971 cpu_manager.go:414] "containerMap: removing container" podUID="4d75d87e-395e-4668-8b7a-42bdd79d08d9" containerName="insights-gathering" Nov 02 20:19:36.225228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.225092 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d75d87e-395e-4668-8b7a-42bdd79d08d9" containerName="insights-gathering" Nov 02 20:19:36.225228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.225106 4971 cpu_manager.go:414] "containerMap: removing container" podUID="02eded99-edf7-41a0-8866-549252be5fe0" containerName="registry" Nov 02 20:19:36.225228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.225113 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02eded99-edf7-41a0-8866-549252be5fe0" containerName="registry" Nov 02 20:19:36.225228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.225126 4971 cpu_manager.go:414] "containerMap: removing container" podUID="9cb81e57-c7dc-4447-959f-d5496ffee976" containerName="registry" Nov 02 20:19:36.225228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.225131 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb81e57-c7dc-4447-959f-d5496ffee976" containerName="registry" Nov 02 20:19:36.225228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.225196 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb81e57-c7dc-4447-959f-d5496ffee976" containerName="registry" Nov 02 20:19:36.225228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.225206 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d8986b-2bb5-4ad9-88a7-fe81cb1f3183" containerName="console" Nov 02 20:19:36.225228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.225212 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d75d87e-395e-4668-8b7a-42bdd79d08d9" containerName="insights-gathering" Nov 02 20:19:36.225228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.225218 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="02eded99-edf7-41a0-8866-549252be5fe0" containerName="registry" Nov 02 20:19:36.225884 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.225326 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb81e57-c7dc-4447-959f-d5496ffee976" containerName="registry" Nov 02 20:19:36.227656 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.227638 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" Nov 02 20:19:36.277417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.277386 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 20:19:36.277417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.277386 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 20:19:36.285276 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.285231 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:36.324121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.324093 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-dsqpp\"" Nov 02 20:19:36.325572 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.325555 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7"] Nov 02 20:19:36.328265 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.328250 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws"] Nov 02 20:19:36.328346 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.328335 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" Nov 02 20:19:36.372205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.372175 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7"] Nov 02 20:19:36.379474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.379456 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-rhhns\"" Nov 02 20:19:36.393472 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.393444 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn"] Nov 02 20:19:36.395983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.395967 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.419846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.419817 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11776af4-4073-4f0e-b03f-083606e60506-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws\" (UID: \"11776af4-4073-4f0e-b03f-083606e60506\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" Nov 02 20:19:36.420012 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.419851 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fdk6\" (UniqueName: \"kubernetes.io/projected/11776af4-4073-4f0e-b03f-083606e60506-kube-api-access-4fdk6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws\" (UID: \"11776af4-4073-4f0e-b03f-083606e60506\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" Nov 02 20:19:36.420012 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.419873 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11776af4-4073-4f0e-b03f-083606e60506-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws\" (UID: \"11776af4-4073-4f0e-b03f-083606e60506\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" Nov 02 20:19:36.445838 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.445816 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 20:19:36.456542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.456517 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn"] Nov 02 20:19:36.480155 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.480073 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-dqj4j\"" Nov 02 20:19:36.520747 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.520722 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86e9d037-889d-47da-ad79-bc2c43104eef-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7\" (UID: \"86e9d037-889d-47da-ad79-bc2c43104eef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" Nov 02 20:19:36.520889 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.520759 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.520889 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.520788 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-272ww\" (UniqueName: \"kubernetes.io/projected/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-kube-api-access-272ww\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.520987 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.520891 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86e9d037-889d-47da-ad79-bc2c43104eef-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7\" (UID: \"86e9d037-889d-47da-ad79-bc2c43104eef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" Nov 02 20:19:36.520987 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.520926 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.520987 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.520950 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.521111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.521025 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11776af4-4073-4f0e-b03f-083606e60506-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws\" (UID: \"11776af4-4073-4f0e-b03f-083606e60506\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" Nov 02 20:19:36.521111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.521062 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fdk6\" (UniqueName: \"kubernetes.io/projected/11776af4-4073-4f0e-b03f-083606e60506-kube-api-access-4fdk6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws\" (UID: \"11776af4-4073-4f0e-b03f-083606e60506\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" Nov 02 20:19:36.521111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.521091 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.521236 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.521114 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.521236 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.521138 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11776af4-4073-4f0e-b03f-083606e60506-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws\" (UID: \"11776af4-4073-4f0e-b03f-083606e60506\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" Nov 02 20:19:36.521236 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.521169 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.521388 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.521328 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.521438 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.521398 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.521438 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.521425 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11776af4-4073-4f0e-b03f-083606e60506-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws\" (UID: \"11776af4-4073-4f0e-b03f-083606e60506\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" Nov 02 20:19:36.521519 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.521434 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8pfj\" (UniqueName: \"kubernetes.io/projected/86e9d037-889d-47da-ad79-bc2c43104eef-kube-api-access-v8pfj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7\" (UID: \"86e9d037-889d-47da-ad79-bc2c43104eef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" Nov 02 20:19:36.521519 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.521467 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11776af4-4073-4f0e-b03f-083606e60506-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws\" (UID: \"11776af4-4073-4f0e-b03f-083606e60506\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" Nov 02 20:19:36.581720 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.581692 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fdk6\" (UniqueName: \"kubernetes.io/projected/11776af4-4073-4f0e-b03f-083606e60506-kube-api-access-4fdk6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws\" (UID: \"11776af4-4073-4f0e-b03f-083606e60506\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" Nov 02 20:19:36.602991 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.602965 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:36.622072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622048 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86e9d037-889d-47da-ad79-bc2c43104eef-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7\" (UID: \"86e9d037-889d-47da-ad79-bc2c43104eef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" Nov 02 20:19:36.622182 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622077 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.622182 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622118 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.622182 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622143 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.622182 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622162 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.622424 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622190 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.622424 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622227 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.622532 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622424 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.622532 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622477 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8pfj\" (UniqueName: \"kubernetes.io/projected/86e9d037-889d-47da-ad79-bc2c43104eef-kube-api-access-v8pfj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7\" (UID: \"86e9d037-889d-47da-ad79-bc2c43104eef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" Nov 02 20:19:36.622532 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622492 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86e9d037-889d-47da-ad79-bc2c43104eef-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7\" (UID: \"86e9d037-889d-47da-ad79-bc2c43104eef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" Nov 02 20:19:36.622532 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622492 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.622532 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622519 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86e9d037-889d-47da-ad79-bc2c43104eef-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7\" (UID: \"86e9d037-889d-47da-ad79-bc2c43104eef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" Nov 02 20:19:36.622789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622541 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.622789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622568 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.622789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622591 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-272ww\" (UniqueName: \"kubernetes.io/projected/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-kube-api-access-272ww\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.622789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622784 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86e9d037-889d-47da-ad79-bc2c43104eef-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7\" (UID: \"86e9d037-889d-47da-ad79-bc2c43104eef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" Nov 02 20:19:36.622973 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622904 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.623013 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.622979 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.623123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.623102 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.624950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.624931 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.625016 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.624995 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.681600 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.681573 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.681896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.681876 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8pfj\" (UniqueName: \"kubernetes.io/projected/86e9d037-889d-47da-ad79-bc2c43104eef-kube-api-access-v8pfj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7\" (UID: \"86e9d037-889d-47da-ad79-bc2c43104eef\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" Nov 02 20:19:36.681964 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.681942 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-272ww\" (UniqueName: \"kubernetes.io/projected/37ff3ea1-40c2-4be8-82bf-af4ae749ebc0-kube-api-access-272ww\") pod \"openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn\" (UID: \"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.707743 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.707723 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:36.841835 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.841807 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" Nov 02 20:19:36.850928 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.850903 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:36.943552 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:36.943525 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" Nov 02 20:19:37.120857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:37.120837 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn"] Nov 02 20:19:37.122611 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:19:37.122585 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ff3ea1_40c2_4be8_82bf_af4ae749ebc0.slice/crio-a5a404331622c3a1a2398a52152abcf217d9057786b38cb6d1ac91d2639448a9 WatchSource:0}: Error finding container a5a404331622c3a1a2398a52152abcf217d9057786b38cb6d1ac91d2639448a9: Status 404 returned error can't find the container with id a5a404331622c3a1a2398a52152abcf217d9057786b38cb6d1ac91d2639448a9 Nov 02 20:19:37.124612 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:37.124596 4971 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:19:37.211867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:37.211841 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws"] Nov 02 20:19:37.213872 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:19:37.213843 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11776af4_4073_4f0e_b03f_083606e60506.slice/crio-89e1722b6987688cd86d0833f8adc42154ffc3a549d1ae9bcbb92f9f54a63a6a WatchSource:0}: Error finding container 89e1722b6987688cd86d0833f8adc42154ffc3a549d1ae9bcbb92f9f54a63a6a: Status 404 returned error can't find the container with id 89e1722b6987688cd86d0833f8adc42154ffc3a549d1ae9bcbb92f9f54a63a6a Nov 02 20:19:37.260751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:37.260722 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" event={"ID":"11776af4-4073-4f0e-b03f-083606e60506","Type":"ContainerStarted","Data":"89e1722b6987688cd86d0833f8adc42154ffc3a549d1ae9bcbb92f9f54a63a6a"} Nov 02 20:19:37.261874 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:37.261852 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" event={"ID":"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0","Type":"ContainerStarted","Data":"a5a404331622c3a1a2398a52152abcf217d9057786b38cb6d1ac91d2639448a9"} Nov 02 20:19:37.305360 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:37.305336 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7"] Nov 02 20:19:37.307879 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:19:37.307851 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e9d037_889d_47da_ad79_bc2c43104eef.slice/crio-7f07c205b3fb956c7e67099197d87f2da9f436d0e937b2945f0eaf225fefb877 WatchSource:0}: Error finding container 7f07c205b3fb956c7e67099197d87f2da9f436d0e937b2945f0eaf225fefb877: Status 404 returned error can't find the container with id 7f07c205b3fb956c7e67099197d87f2da9f436d0e937b2945f0eaf225fefb877 Nov 02 20:19:38.273441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:38.273403 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" event={"ID":"86e9d037-889d-47da-ad79-bc2c43104eef","Type":"ContainerStarted","Data":"7f07c205b3fb956c7e67099197d87f2da9f436d0e937b2945f0eaf225fefb877"} Nov 02 20:19:40.294013 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:40.293975 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:19:42.686969 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:42.686914 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:19:42.687454 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:42.687015 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:19:42.687454 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:42.687058 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:19:43.302911 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:43.302868 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" event={"ID":"37ff3ea1-40c2-4be8-82bf-af4ae749ebc0","Type":"ContainerStarted","Data":"ff985dbb4a666bd27621f034412956fe1b46384fc406fad3a4d5b138fd103e6e"} Nov 02 20:19:43.708328 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:43.708239 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:43.714316 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:43.714273 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:43.829865 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:43.829809 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" podStartSLOduration=2.268557419 podStartE2EDuration="7.829796138s" totalImagesPullingTime="5.561238719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:43.464231338 +0000 UTC m=+311.753887865" watchObservedRunningTime="2026-11-02 20:19:43.829796138 +0000 UTC m=+312.119452660" Nov 02 20:19:44.308389 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:44.308358 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:44.309664 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:44.309631 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn" Nov 02 20:19:45.314557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:45.314519 4971 generic.go:356] "Generic (PLEG): container finished" podID="86e9d037-889d-47da-ad79-bc2c43104eef" containerID="f1bf1a8b6d467b16ad5ec6971801382c22e1a40a6beb3e4b7a462ce86944419d" exitCode=0 Nov 02 20:19:45.315065 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:45.314567 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" event={"ID":"86e9d037-889d-47da-ad79-bc2c43104eef","Type":"ContainerDied","Data":"f1bf1a8b6d467b16ad5ec6971801382c22e1a40a6beb3e4b7a462ce86944419d"} Nov 02 20:19:45.316732 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:45.316709 4971 generic.go:356] "Generic (PLEG): container finished" podID="11776af4-4073-4f0e-b03f-083606e60506" containerID="770d27726554f96766ed31e31afef53b00d3e0ac0d17f04d6c41cb395b205840" exitCode=0 Nov 02 20:19:45.316943 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:45.316833 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" event={"ID":"11776af4-4073-4f0e-b03f-083606e60506","Type":"ContainerDied","Data":"770d27726554f96766ed31e31afef53b00d3e0ac0d17f04d6c41cb395b205840"} Nov 02 20:19:47.328075 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:47.328049 4971 generic.go:356] "Generic (PLEG): container finished" podID="86e9d037-889d-47da-ad79-bc2c43104eef" containerID="f8c6a7f0ef576617b115af4540a431af81aa881fe2fddb6006d7c42a5758dd90" exitCode=0 Nov 02 20:19:47.328382 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:47.328127 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" event={"ID":"86e9d037-889d-47da-ad79-bc2c43104eef","Type":"ContainerDied","Data":"f8c6a7f0ef576617b115af4540a431af81aa881fe2fddb6006d7c42a5758dd90"} Nov 02 20:19:48.335223 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:48.335187 4971 generic.go:356] "Generic (PLEG): container finished" podID="11776af4-4073-4f0e-b03f-083606e60506" containerID="c9c4e794881192350083daa12b3bc35d0e6361a6f2dbbaaa01db6b84dc944d06" exitCode=0 Nov 02 20:19:48.335679 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:48.335266 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" event={"ID":"11776af4-4073-4f0e-b03f-083606e60506","Type":"ContainerDied","Data":"c9c4e794881192350083daa12b3bc35d0e6361a6f2dbbaaa01db6b84dc944d06"} Nov 02 20:19:54.374223 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:54.374190 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" event={"ID":"86e9d037-889d-47da-ad79-bc2c43104eef","Type":"ContainerStarted","Data":"77b47b9f06c458ce2c2aba611a3954f09301f2a8e3d6ab8c3859270e268c5418"} Nov 02 20:19:54.487104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:54.486989 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" podStartSLOduration=2.609961773 podStartE2EDuration="18.48697551s" totalImagesPullingTime="15.877013737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:54.484981366 +0000 UTC m=+322.774637893" watchObservedRunningTime="2026-11-02 20:19:54.48697551 +0000 UTC m=+322.776632038" Nov 02 20:19:55.381494 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:55.381460 4971 generic.go:356] "Generic (PLEG): container finished" podID="86e9d037-889d-47da-ad79-bc2c43104eef" containerID="77b47b9f06c458ce2c2aba611a3954f09301f2a8e3d6ab8c3859270e268c5418" exitCode=0 Nov 02 20:19:55.381929 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:55.381513 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" event={"ID":"86e9d037-889d-47da-ad79-bc2c43104eef","Type":"ContainerDied","Data":"77b47b9f06c458ce2c2aba611a3954f09301f2a8e3d6ab8c3859270e268c5418"} Nov 02 20:19:56.387947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:56.387911 4971 generic.go:356] "Generic (PLEG): container finished" podID="11776af4-4073-4f0e-b03f-083606e60506" containerID="e0056cb521e67f73db6d0b7d8b579405186ffbf1f304f323f25efdaa81f17d6f" exitCode=0 Nov 02 20:19:56.388317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:56.387989 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" event={"ID":"11776af4-4073-4f0e-b03f-083606e60506","Type":"ContainerDied","Data":"e0056cb521e67f73db6d0b7d8b579405186ffbf1f304f323f25efdaa81f17d6f"} Nov 02 20:19:56.522256 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:56.522224 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" Nov 02 20:19:56.588914 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:56.588891 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e9d037-889d-47da-ad79-bc2c43104eef-util\" (UniqueName: \"kubernetes.io/empty-dir/86e9d037-889d-47da-ad79-bc2c43104eef-util\") pod \"86e9d037-889d-47da-ad79-bc2c43104eef\" (UID: \"86e9d037-889d-47da-ad79-bc2c43104eef\") " Nov 02 20:19:56.589042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:56.588966 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e9d037-889d-47da-ad79-bc2c43104eef-kube-api-access-v8pfj\" (UniqueName: \"kubernetes.io/projected/86e9d037-889d-47da-ad79-bc2c43104eef-kube-api-access-v8pfj\") pod \"86e9d037-889d-47da-ad79-bc2c43104eef\" (UID: \"86e9d037-889d-47da-ad79-bc2c43104eef\") " Nov 02 20:19:56.589042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:56.589002 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e9d037-889d-47da-ad79-bc2c43104eef-bundle\" (UniqueName: \"kubernetes.io/empty-dir/86e9d037-889d-47da-ad79-bc2c43104eef-bundle\") pod \"86e9d037-889d-47da-ad79-bc2c43104eef\" (UID: \"86e9d037-889d-47da-ad79-bc2c43104eef\") " Nov 02 20:19:56.589669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:56.589645 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e9d037-889d-47da-ad79-bc2c43104eef-bundle" pod "86e9d037-889d-47da-ad79-bc2c43104eef" (UID: "86e9d037-889d-47da-ad79-bc2c43104eef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:56.591366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:56.591329 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e9d037-889d-47da-ad79-bc2c43104eef-kube-api-access-v8pfj" pod "86e9d037-889d-47da-ad79-bc2c43104eef" (UID: "86e9d037-889d-47da-ad79-bc2c43104eef"). InnerVolumeSpecName "kube-api-access-v8pfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:56.595294 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:56.595264 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e9d037-889d-47da-ad79-bc2c43104eef-util" pod "86e9d037-889d-47da-ad79-bc2c43104eef" (UID: "86e9d037-889d-47da-ad79-bc2c43104eef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:56.689998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:56.689973 4971 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86e9d037-889d-47da-ad79-bc2c43104eef-util\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:56.690145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:56.690014 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8pfj\" (UniqueName: \"kubernetes.io/projected/86e9d037-889d-47da-ad79-bc2c43104eef-kube-api-access-v8pfj\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:56.690145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:56.690024 4971 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86e9d037-889d-47da-ad79-bc2c43104eef-bundle\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:57.394045 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:57.394012 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" Nov 02 20:19:57.394045 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:57.394037 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afkmh7" event={"ID":"86e9d037-889d-47da-ad79-bc2c43104eef","Type":"ContainerDied","Data":"7f07c205b3fb956c7e67099197d87f2da9f436d0e937b2945f0eaf225fefb877"} Nov 02 20:19:57.394576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:57.394098 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f07c205b3fb956c7e67099197d87f2da9f436d0e937b2945f0eaf225fefb877" Nov 02 20:19:57.519660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:57.519637 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" Nov 02 20:19:57.597190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:57.597165 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11776af4-4073-4f0e-b03f-083606e60506-kube-api-access-4fdk6\" (UniqueName: \"kubernetes.io/projected/11776af4-4073-4f0e-b03f-083606e60506-kube-api-access-4fdk6\") pod \"11776af4-4073-4f0e-b03f-083606e60506\" (UID: \"11776af4-4073-4f0e-b03f-083606e60506\") " Nov 02 20:19:57.597306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:57.597219 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11776af4-4073-4f0e-b03f-083606e60506-bundle\" (UniqueName: \"kubernetes.io/empty-dir/11776af4-4073-4f0e-b03f-083606e60506-bundle\") pod \"11776af4-4073-4f0e-b03f-083606e60506\" (UID: \"11776af4-4073-4f0e-b03f-083606e60506\") " Nov 02 20:19:57.597306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:57.597242 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11776af4-4073-4f0e-b03f-083606e60506-util\" (UniqueName: \"kubernetes.io/empty-dir/11776af4-4073-4f0e-b03f-083606e60506-util\") pod \"11776af4-4073-4f0e-b03f-083606e60506\" (UID: \"11776af4-4073-4f0e-b03f-083606e60506\") " Nov 02 20:19:57.597719 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:57.597694 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11776af4-4073-4f0e-b03f-083606e60506-bundle" pod "11776af4-4073-4f0e-b03f-083606e60506" (UID: "11776af4-4073-4f0e-b03f-083606e60506"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:57.599364 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:57.599343 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11776af4-4073-4f0e-b03f-083606e60506-kube-api-access-4fdk6" pod "11776af4-4073-4f0e-b03f-083606e60506" (UID: "11776af4-4073-4f0e-b03f-083606e60506"). InnerVolumeSpecName "kube-api-access-4fdk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:57.601525 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:57.601505 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11776af4-4073-4f0e-b03f-083606e60506-util" pod "11776af4-4073-4f0e-b03f-083606e60506" (UID: "11776af4-4073-4f0e-b03f-083606e60506"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:57.698264 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:57.698190 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fdk6\" (UniqueName: \"kubernetes.io/projected/11776af4-4073-4f0e-b03f-083606e60506-kube-api-access-4fdk6\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:57.698264 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:57.698215 4971 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11776af4-4073-4f0e-b03f-083606e60506-bundle\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:57.698264 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:57.698230 4971 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11776af4-4073-4f0e-b03f-083606e60506-util\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:58.401075 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:58.401048 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" Nov 02 20:19:58.401075 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:58.401059 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d767tws" event={"ID":"11776af4-4073-4f0e-b03f-083606e60506","Type":"ContainerDied","Data":"89e1722b6987688cd86d0833f8adc42154ffc3a549d1ae9bcbb92f9f54a63a6a"} Nov 02 20:19:58.401500 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:19:58.401127 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89e1722b6987688cd86d0833f8adc42154ffc3a549d1ae9bcbb92f9f54a63a6a" Nov 02 20:20:03.336928 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:20:03.336894 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:20:13.460089 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:13.460042 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:13.713291 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:13.713201 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:17.745091 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:17.745056 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:18.026176 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.026093 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:18.791979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.791944 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:18.905841 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.905810 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2"] Nov 02 20:20:18.906163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.906149 4971 cpu_manager.go:414] "containerMap: removing container" podUID="11776af4-4073-4f0e-b03f-083606e60506" containerName="util" Nov 02 20:20:18.906211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.906164 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11776af4-4073-4f0e-b03f-083606e60506" containerName="util" Nov 02 20:20:18.906211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.906173 4971 cpu_manager.go:414] "containerMap: removing container" podUID="11776af4-4073-4f0e-b03f-083606e60506" containerName="extract" Nov 02 20:20:18.906211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.906178 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11776af4-4073-4f0e-b03f-083606e60506" containerName="extract" Nov 02 20:20:18.906211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.906201 4971 cpu_manager.go:414] "containerMap: removing container" podUID="86e9d037-889d-47da-ad79-bc2c43104eef" containerName="util" Nov 02 20:20:18.906211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.906206 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e9d037-889d-47da-ad79-bc2c43104eef" containerName="util" Nov 02 20:20:18.906369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.906219 4971 cpu_manager.go:414] "containerMap: removing container" podUID="86e9d037-889d-47da-ad79-bc2c43104eef" containerName="extract" Nov 02 20:20:18.906369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.906223 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e9d037-889d-47da-ad79-bc2c43104eef" containerName="extract" Nov 02 20:20:18.906369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.906229 4971 cpu_manager.go:414] "containerMap: removing container" podUID="86e9d037-889d-47da-ad79-bc2c43104eef" containerName="pull" Nov 02 20:20:18.906369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.906233 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e9d037-889d-47da-ad79-bc2c43104eef" containerName="pull" Nov 02 20:20:18.906369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.906247 4971 cpu_manager.go:414] "containerMap: removing container" podUID="11776af4-4073-4f0e-b03f-083606e60506" containerName="pull" Nov 02 20:20:18.906369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.906253 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11776af4-4073-4f0e-b03f-083606e60506" containerName="pull" Nov 02 20:20:18.906369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.906306 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e9d037-889d-47da-ad79-bc2c43104eef" containerName="extract" Nov 02 20:20:18.906369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.906315 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="11776af4-4073-4f0e-b03f-083606e60506" containerName="extract" Nov 02 20:20:18.908886 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.908869 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:18.958483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.958458 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 20:20:18.958483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.958473 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 20:20:18.958624 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.958512 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 20:20:18.958712 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.958696 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 20:20:18.958780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:18.958764 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 20:20:19.005367 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.005344 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-72vcs\"" Nov 02 20:20:19.019615 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.019592 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2"] Nov 02 20:20:19.054950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.054892 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fcabb4b-edb9-43fb-857b-b90309bce6f5-metrics-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.054950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.054924 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fcabb4b-edb9-43fb-857b-b90309bce6f5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.054950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.054950 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fcabb4b-edb9-43fb-857b-b90309bce6f5-manifests\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.055202 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.055028 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trm2\" (UniqueName: \"kubernetes.io/projected/7fcabb4b-edb9-43fb-857b-b90309bce6f5-kube-api-access-9trm2\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.055261 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.055233 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fcabb4b-edb9-43fb-857b-b90309bce6f5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.055316 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.055272 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fcabb4b-edb9-43fb-857b-b90309bce6f5-tmp\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.092477 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.092452 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:19.155872 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.155853 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fcabb4b-edb9-43fb-857b-b90309bce6f5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.155988 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.155881 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fcabb4b-edb9-43fb-857b-b90309bce6f5-tmp\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.155988 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.155907 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fcabb4b-edb9-43fb-857b-b90309bce6f5-metrics-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.155988 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.155926 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fcabb4b-edb9-43fb-857b-b90309bce6f5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.156180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.156051 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fcabb4b-edb9-43fb-857b-b90309bce6f5-manifests\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.156180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.156107 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trm2\" (UniqueName: \"kubernetes.io/projected/7fcabb4b-edb9-43fb-857b-b90309bce6f5-kube-api-access-9trm2\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.156482 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.156454 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7fcabb4b-edb9-43fb-857b-b90309bce6f5-manifests\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.158630 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.158604 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fcabb4b-edb9-43fb-857b-b90309bce6f5-tmp\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.158733 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.158672 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fcabb4b-edb9-43fb-857b-b90309bce6f5-metrics-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.158802 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.158761 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7fcabb4b-edb9-43fb-857b-b90309bce6f5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.158843 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.158826 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fcabb4b-edb9-43fb-857b-b90309bce6f5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.214160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.214130 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trm2\" (UniqueName: \"kubernetes.io/projected/7fcabb4b-edb9-43fb-857b-b90309bce6f5-kube-api-access-9trm2\") pod \"opendatahub-operator-controller-manager-5984b9787d-fnpr2\" (UID: \"7fcabb4b-edb9-43fb-857b-b90309bce6f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.221801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.221782 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:19.341388 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.341318 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:19.626670 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.626619 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:19.693435 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.693413 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2"] Nov 02 20:20:19.694999 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:20:19.694975 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcabb4b_edb9_43fb_857b_b90309bce6f5.slice/crio-a8e6cd875b4bbabb8db9bfb0c26b133ee77a2738ac8fe31f5b14d21b0e8089eb WatchSource:0}: Error finding container a8e6cd875b4bbabb8db9bfb0c26b133ee77a2738ac8fe31f5b14d21b0e8089eb: Status 404 returned error can't find the container with id a8e6cd875b4bbabb8db9bfb0c26b133ee77a2738ac8fe31f5b14d21b0e8089eb Nov 02 20:20:19.871532 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:19.871503 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:20.494488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:20.494459 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" event={"ID":"7fcabb4b-edb9-43fb-857b-b90309bce6f5","Type":"ContainerStarted","Data":"a8e6cd875b4bbabb8db9bfb0c26b133ee77a2738ac8fe31f5b14d21b0e8089eb"} Nov 02 20:20:22.772282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:22.772245 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:23.509066 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:23.509038 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" event={"ID":"7fcabb4b-edb9-43fb-857b-b90309bce6f5","Type":"ContainerStarted","Data":"990e9288cb62bdfccdf4d8185af263ae7e69a130a6c30c0e20b5299d8b17e526"} Nov 02 20:20:23.546439 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:23.546408 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:24.514997 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:24.514960 4971 generic.go:356] "Generic (PLEG): container finished" podID="7fcabb4b-edb9-43fb-857b-b90309bce6f5" containerID="990e9288cb62bdfccdf4d8185af263ae7e69a130a6c30c0e20b5299d8b17e526" exitCode=0 Nov 02 20:20:24.515405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:24.515043 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" event={"ID":"7fcabb4b-edb9-43fb-857b-b90309bce6f5","Type":"ContainerDied","Data":"990e9288cb62bdfccdf4d8185af263ae7e69a130a6c30c0e20b5299d8b17e526"} Nov 02 20:20:24.752283 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:24.752255 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:25.037740 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:25.037711 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:26.526094 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:26.526047 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" event={"ID":"7fcabb4b-edb9-43fb-857b-b90309bce6f5","Type":"ContainerStarted","Data":"94f5934414251317896e82c9b05619f7b4ca552dc5ce5038e5a380147875262f"} Nov 02 20:20:26.526505 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:26.526193 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:26.801538 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:26.801426 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" podStartSLOduration=3.737278963 podStartE2EDuration="8.801411689s" totalImagesPullingTime="4.636983082s" totalInitContainerRuntime="427.149644ms" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:26.753053208 +0000 UTC m=+355.042709745" watchObservedRunningTime="2026-11-02 20:20:26.801411689 +0000 UTC m=+355.091068216" Nov 02 20:20:28.141710 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:28.141677 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:29.542794 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:29.542762 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:30.342481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:30.342444 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:30.791442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:30.791404 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:31.158140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:31.158068 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:33.336725 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:20:33.336682 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:20:34.344344 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:34.344307 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:34.585879 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:34.585844 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:36.346726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:36.346687 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq"] Nov 02 20:20:36.352445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:36.352424 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" Nov 02 20:20:36.398015 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:36.397984 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq"] Nov 02 20:20:36.442972 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:36.442945 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 20:20:36.442972 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:36.442953 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 20:20:36.443194 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:36.443117 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-njndk\"" Nov 02 20:20:36.490112 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:36.490066 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zrfj\" (UniqueName: \"kubernetes.io/projected/84becb7d-d0d7-486c-82b1-36e4b1c4fe08-kube-api-access-6zrfj\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m65rq\" (UID: \"84becb7d-d0d7-486c-82b1-36e4b1c4fe08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" Nov 02 20:20:36.490280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:36.490117 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84becb7d-d0d7-486c-82b1-36e4b1c4fe08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m65rq\" (UID: \"84becb7d-d0d7-486c-82b1-36e4b1c4fe08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" Nov 02 20:20:36.591450 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:36.591424 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zrfj\" (UniqueName: \"kubernetes.io/projected/84becb7d-d0d7-486c-82b1-36e4b1c4fe08-kube-api-access-6zrfj\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m65rq\" (UID: \"84becb7d-d0d7-486c-82b1-36e4b1c4fe08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" Nov 02 20:20:36.591569 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:36.591459 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84becb7d-d0d7-486c-82b1-36e4b1c4fe08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m65rq\" (UID: \"84becb7d-d0d7-486c-82b1-36e4b1c4fe08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" Nov 02 20:20:36.591834 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:36.591818 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84becb7d-d0d7-486c-82b1-36e4b1c4fe08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m65rq\" (UID: \"84becb7d-d0d7-486c-82b1-36e4b1c4fe08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" Nov 02 20:20:36.650680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:36.650621 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zrfj\" (UniqueName: \"kubernetes.io/projected/84becb7d-d0d7-486c-82b1-36e4b1c4fe08-kube-api-access-6zrfj\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m65rq\" (UID: \"84becb7d-d0d7-486c-82b1-36e4b1c4fe08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" Nov 02 20:20:36.664263 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:36.664243 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" Nov 02 20:20:36.955425 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:36.955342 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:37.129238 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:37.129213 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq"] Nov 02 20:20:37.131638 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:20:37.131609 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84becb7d_d0d7_486c_82b1_36e4b1c4fe08.slice/crio-695020f7a5a418825c6a5d4ff2317cbbd40cefb5457a8199042675c2cf3adf00 WatchSource:0}: Error finding container 695020f7a5a418825c6a5d4ff2317cbbd40cefb5457a8199042675c2cf3adf00: Status 404 returned error can't find the container with id 695020f7a5a418825c6a5d4ff2317cbbd40cefb5457a8199042675c2cf3adf00 Nov 02 20:20:37.202231 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:37.202200 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:37.532984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:37.532955 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-fnpr2" Nov 02 20:20:37.573580 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:37.573552 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" event={"ID":"84becb7d-d0d7-486c-82b1-36e4b1c4fe08","Type":"ContainerStarted","Data":"695020f7a5a418825c6a5d4ff2317cbbd40cefb5457a8199042675c2cf3adf00"} Nov 02 20:20:37.801144 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:37.801047 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:38.057789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:38.057713 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:39.495024 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:39.494991 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-8527l"] Nov 02 20:20:39.498684 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:39.498654 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8527l" Nov 02 20:20:39.549423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:39.549393 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 02 20:20:39.549592 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:39.549497 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5pkfg\"" Nov 02 20:20:39.598152 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:39.598125 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-8527l"] Nov 02 20:20:39.621286 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:39.621253 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjvx8\" (UniqueName: \"kubernetes.io/projected/8a6c5615-a45d-40bf-a881-b1f7c32c0ffd-kube-api-access-rjvx8\") pod \"dns-operator-controller-manager-8557cbdb84-8527l\" (UID: \"8a6c5615-a45d-40bf-a881-b1f7c32c0ffd\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8527l" Nov 02 20:20:39.722306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:39.722277 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjvx8\" (UniqueName: \"kubernetes.io/projected/8a6c5615-a45d-40bf-a881-b1f7c32c0ffd-kube-api-access-rjvx8\") pod \"dns-operator-controller-manager-8557cbdb84-8527l\" (UID: \"8a6c5615-a45d-40bf-a881-b1f7c32c0ffd\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8527l" Nov 02 20:20:39.784806 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:39.784734 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjvx8\" (UniqueName: \"kubernetes.io/projected/8a6c5615-a45d-40bf-a881-b1f7c32c0ffd-kube-api-access-rjvx8\") pod \"dns-operator-controller-manager-8557cbdb84-8527l\" (UID: \"8a6c5615-a45d-40bf-a881-b1f7c32c0ffd\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8527l" Nov 02 20:20:39.814951 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:39.814925 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8527l" Nov 02 20:20:39.935608 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:39.935568 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:40.197120 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:40.196734 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-8527l"] Nov 02 20:20:40.200839 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:20:40.200815 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6c5615_a45d_40bf_a881_b1f7c32c0ffd.slice/crio-f623a353428447d07147fb0f91bc0d242fa02422863db54c1716084ea147ed1a WatchSource:0}: Error finding container f623a353428447d07147fb0f91bc0d242fa02422863db54c1716084ea147ed1a: Status 404 returned error can't find the container with id f623a353428447d07147fb0f91bc0d242fa02422863db54c1716084ea147ed1a Nov 02 20:20:40.258446 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:40.258416 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:40.589797 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:40.589725 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8527l" event={"ID":"8a6c5615-a45d-40bf-a881-b1f7c32c0ffd","Type":"ContainerStarted","Data":"f623a353428447d07147fb0f91bc0d242fa02422863db54c1716084ea147ed1a"} Nov 02 20:20:40.617211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:40.617179 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:40.905142 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:40.905110 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:41.220384 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:41.220291 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:43.357603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:43.357557 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:43.612817 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:43.612737 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:43.872580 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:43.872463 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:44.609110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:44.609066 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8527l" event={"ID":"8a6c5615-a45d-40bf-a881-b1f7c32c0ffd","Type":"ContainerStarted","Data":"8eb34025b60779fabe520d81f7fd4923ae6d84375df2b9f5685064556457a301"} Nov 02 20:20:44.609392 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:44.609202 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8527l" Nov 02 20:20:44.707070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:44.707043 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-wztv4"] Nov 02 20:20:44.710622 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:44.710605 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wztv4" Nov 02 20:20:44.734651 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:44.734605 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8527l" podStartSLOduration=1.747942801 podStartE2EDuration="5.734593505s" totalImagesPullingTime="3.986650704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:44.734204156 +0000 UTC m=+373.023860683" watchObservedRunningTime="2026-11-02 20:20:44.734593505 +0000 UTC m=+373.024250038" Nov 02 20:20:44.765660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:44.765637 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-wztv4"] Nov 02 20:20:44.828447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:44.828391 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-cft2z\"" Nov 02 20:20:44.868305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:44.868279 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvgc2\" (UniqueName: \"kubernetes.io/projected/bcf7c59a-ef20-4eea-893d-64595f68ff8d-kube-api-access-tvgc2\") pod \"authorino-operator-7b7bb4bc77-wztv4\" (UID: \"bcf7c59a-ef20-4eea-893d-64595f68ff8d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-wztv4" Nov 02 20:20:44.969623 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:44.969597 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvgc2\" (UniqueName: \"kubernetes.io/projected/bcf7c59a-ef20-4eea-893d-64595f68ff8d-kube-api-access-tvgc2\") pod \"authorino-operator-7b7bb4bc77-wztv4\" (UID: \"bcf7c59a-ef20-4eea-893d-64595f68ff8d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-wztv4" Nov 02 20:20:45.027672 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:45.027640 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvgc2\" (UniqueName: \"kubernetes.io/projected/bcf7c59a-ef20-4eea-893d-64595f68ff8d-kube-api-access-tvgc2\") pod \"authorino-operator-7b7bb4bc77-wztv4\" (UID: \"bcf7c59a-ef20-4eea-893d-64595f68ff8d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-wztv4" Nov 02 20:20:45.323600 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:45.323568 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wztv4" Nov 02 20:20:45.426833 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:45.426805 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:45.615506 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:45.615475 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" event={"ID":"84becb7d-d0d7-486c-82b1-36e4b1c4fe08","Type":"ContainerStarted","Data":"8de81e1c2b8feaccad6facd782cbfa6876eab287e231d56da85ff0815ad923bd"} Nov 02 20:20:45.746101 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:45.746043 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" podStartSLOduration=2.305795092 podStartE2EDuration="9.746030635s" totalImagesPullingTime="7.440235543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:45.745614065 +0000 UTC m=+374.035270593" watchObservedRunningTime="2026-11-02 20:20:45.746030635 +0000 UTC m=+374.035687157" Nov 02 20:20:45.794256 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:45.794225 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-wztv4"] Nov 02 20:20:45.796027 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:20:45.796002 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf7c59a_ef20_4eea_893d_64595f68ff8d.slice/crio-47c4d29dc9f7eb19f5908a33aad86d15f661eee6d9e511e6f9d9aa6250b30155 WatchSource:0}: Error finding container 47c4d29dc9f7eb19f5908a33aad86d15f661eee6d9e511e6f9d9aa6250b30155: Status 404 returned error can't find the container with id 47c4d29dc9f7eb19f5908a33aad86d15f661eee6d9e511e6f9d9aa6250b30155 Nov 02 20:20:45.815501 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:45.815480 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:46.621332 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:46.621289 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wztv4" event={"ID":"bcf7c59a-ef20-4eea-893d-64595f68ff8d","Type":"ContainerStarted","Data":"47c4d29dc9f7eb19f5908a33aad86d15f661eee6d9e511e6f9d9aa6250b30155"} Nov 02 20:20:46.621823 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:46.621450 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" Nov 02 20:20:46.953354 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:46.953275 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:47.199410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:47.199372 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:47.512025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:47.511997 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:48.973717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:48.973685 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:49.222484 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:49.222455 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:49.637189 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:49.637156 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wztv4" event={"ID":"bcf7c59a-ef20-4eea-893d-64595f68ff8d","Type":"ContainerStarted","Data":"b1e0f13e55d1c7c0da5a1d94b6e01fc7fda8c04eb86dac52f75f043eed64dcd9"} Nov 02 20:20:49.637351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:49.637286 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wztv4" Nov 02 20:20:49.749201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:49.749138 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wztv4" podStartSLOduration=2.761898013 podStartE2EDuration="5.749121085s" totalImagesPullingTime="2.987223072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:49.746959864 +0000 UTC m=+378.036616392" watchObservedRunningTime="2026-11-02 20:20:49.749121085 +0000 UTC m=+378.038777613" Nov 02 20:20:49.962280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:49.962198 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:50.413157 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:50.413122 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:51.231411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:51.231376 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:51.480200 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:51.480170 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:51.900002 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:51.899969 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:52.146754 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:52.146721 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:52.395416 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:52.395385 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:52.946133 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:52.946105 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:53.238376 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.238279 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:53.483735 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.483688 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:53.596230 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.596159 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4"] Nov 02 20:20:53.599744 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.599728 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.636576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.636066 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232e2115-ae5c-4105-8b4f-f74061831f82-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.636576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.636122 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232e2115-ae5c-4105-8b4f-f74061831f82-istio-token\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.636576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.636154 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232e2115-ae5c-4105-8b4f-f74061831f82-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.636576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.636186 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232e2115-ae5c-4105-8b4f-f74061831f82-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.636576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.636216 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232e2115-ae5c-4105-8b4f-f74061831f82-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.636576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.636244 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232e2115-ae5c-4105-8b4f-f74061831f82-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.636576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.636267 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml898\" (UniqueName: \"kubernetes.io/projected/232e2115-ae5c-4105-8b4f-f74061831f82-kube-api-access-ml898\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.636576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.636368 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232e2115-ae5c-4105-8b4f-f74061831f82-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.636576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.636398 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232e2115-ae5c-4105-8b4f-f74061831f82-istio-data\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.677940 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.677917 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4"] Nov 02 20:20:53.737014 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.736986 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-56rbs\"" Nov 02 20:20:53.737014 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.737004 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232e2115-ae5c-4105-8b4f-f74061831f82-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.737217 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.737044 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232e2115-ae5c-4105-8b4f-f74061831f82-istio-data\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.737217 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.737131 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232e2115-ae5c-4105-8b4f-f74061831f82-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.737217 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.737158 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232e2115-ae5c-4105-8b4f-f74061831f82-istio-token\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.737217 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.737187 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232e2115-ae5c-4105-8b4f-f74061831f82-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.737419 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.737217 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232e2115-ae5c-4105-8b4f-f74061831f82-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.737419 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.737246 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232e2115-ae5c-4105-8b4f-f74061831f82-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.737419 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.737274 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232e2115-ae5c-4105-8b4f-f74061831f82-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.737419 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.737302 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml898\" (UniqueName: \"kubernetes.io/projected/232e2115-ae5c-4105-8b4f-f74061831f82-kube-api-access-ml898\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.737630 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.737608 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232e2115-ae5c-4105-8b4f-f74061831f82-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.737979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.737927 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232e2115-ae5c-4105-8b4f-f74061831f82-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.737979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.737961 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232e2115-ae5c-4105-8b4f-f74061831f82-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.738350 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.738325 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232e2115-ae5c-4105-8b4f-f74061831f82-istio-data\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.738458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.738409 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232e2115-ae5c-4105-8b4f-f74061831f82-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.739628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.739606 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232e2115-ae5c-4105-8b4f-f74061831f82-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.740213 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.740196 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232e2115-ae5c-4105-8b4f-f74061831f82-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.750862 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.750835 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:53.801992 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.801965 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232e2115-ae5c-4105-8b4f-f74061831f82-istio-token\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.802218 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.802196 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml898\" (UniqueName: \"kubernetes.io/projected/232e2115-ae5c-4105-8b4f-f74061831f82-kube-api-access-ml898\") pod \"data-science-gateway-data-science-gateway-class-65844c896fdfxx4\" (UID: \"232e2115-ae5c-4105-8b4f-f74061831f82\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:53.911978 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:53.911949 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:54.001186 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:54.001155 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:54.254421 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:54.254339 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:54.383946 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:54.383922 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4"] Nov 02 20:20:54.386151 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:20:54.386125 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232e2115_ae5c_4105_8b4f_f74061831f82.slice/crio-08cf30b7f64f96b92ef5ef9acc83bf62d9a70800160d1ffb67117a17911da64d WatchSource:0}: Error finding container 08cf30b7f64f96b92ef5ef9acc83bf62d9a70800160d1ffb67117a17911da64d: Status 404 returned error can't find the container with id 08cf30b7f64f96b92ef5ef9acc83bf62d9a70800160d1ffb67117a17911da64d Nov 02 20:20:54.388068 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:54.388035 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:20:54.388163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:54.388115 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:20:54.388163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:54.388144 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:20:54.661642 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:54.661608 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" event={"ID":"232e2115-ae5c-4105-8b4f-f74061831f82","Type":"ContainerStarted","Data":"71e9b45c272f28ff329c6df86bad054510c89270c51433c1727e594623b0ad95"} Nov 02 20:20:54.661819 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:54.661667 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" event={"ID":"232e2115-ae5c-4105-8b4f-f74061831f82","Type":"ContainerStarted","Data":"08cf30b7f64f96b92ef5ef9acc83bf62d9a70800160d1ffb67117a17911da64d"} Nov 02 20:20:54.813458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:54.813409 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" podStartSLOduration=1.813394956 podStartE2EDuration="1.813394956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:54.812258705 +0000 UTC m=+383.101915235" watchObservedRunningTime="2026-11-02 20:20:54.813394956 +0000 UTC m=+383.103051484" Nov 02 20:20:54.912788 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:54.912723 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:55.618390 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:55.618357 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8527l" Nov 02 20:20:55.917041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:55.916974 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:56.060280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.060248 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hmvzj"] Nov 02 20:20:56.063857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.063841 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" Nov 02 20:20:56.081292 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.081261 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:20:56.116571 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.116546 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 20:20:56.116680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.116620 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4x2wt\"" Nov 02 20:20:56.117444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.117425 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hmvzj"] Nov 02 20:20:56.158498 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.158468 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mwtj\" (UniqueName: \"kubernetes.io/projected/3819be82-2b2e-4f1b-937d-5d90305375d0-kube-api-access-4mwtj\") pod \"limitador-limitador-74dfcfb894-hmvzj\" (UID: \"3819be82-2b2e-4f1b-937d-5d90305375d0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" Nov 02 20:20:56.158608 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.158511 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3819be82-2b2e-4f1b-937d-5d90305375d0-config-file\") pod \"limitador-limitador-74dfcfb894-hmvzj\" (UID: \"3819be82-2b2e-4f1b-937d-5d90305375d0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" Nov 02 20:20:56.259868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.259812 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mwtj\" (UniqueName: \"kubernetes.io/projected/3819be82-2b2e-4f1b-937d-5d90305375d0-kube-api-access-4mwtj\") pod \"limitador-limitador-74dfcfb894-hmvzj\" (UID: \"3819be82-2b2e-4f1b-937d-5d90305375d0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" Nov 02 20:20:56.259868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.259849 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3819be82-2b2e-4f1b-937d-5d90305375d0-config-file\") pod \"limitador-limitador-74dfcfb894-hmvzj\" (UID: \"3819be82-2b2e-4f1b-937d-5d90305375d0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" Nov 02 20:20:56.260450 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.260433 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3819be82-2b2e-4f1b-937d-5d90305375d0-config-file\") pod \"limitador-limitador-74dfcfb894-hmvzj\" (UID: \"3819be82-2b2e-4f1b-937d-5d90305375d0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" Nov 02 20:20:56.311955 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.311933 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hmvzj"] Nov 02 20:20:56.317918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.317899 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mwtj\" (UniqueName: \"kubernetes.io/projected/3819be82-2b2e-4f1b-937d-5d90305375d0-kube-api-access-4mwtj\") pod \"limitador-limitador-74dfcfb894-hmvzj\" (UID: \"3819be82-2b2e-4f1b-937d-5d90305375d0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" Nov 02 20:20:56.376637 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.376616 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" Nov 02 20:20:56.670402 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.670371 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:56.671403 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.671383 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-65844c896fdfxx4" Nov 02 20:20:56.744111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:56.744076 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hmvzj"] Nov 02 20:20:56.746127 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:20:56.746099 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3819be82_2b2e_4f1b_937d_5d90305375d0.slice/crio-80bbd5cf3009b2e21ef1e18dc5791a94a301e8d79a3ae7e77119da934290d254 WatchSource:0}: Error finding container 80bbd5cf3009b2e21ef1e18dc5791a94a301e8d79a3ae7e77119da934290d254: Status 404 returned error can't find the container with id 80bbd5cf3009b2e21ef1e18dc5791a94a301e8d79a3ae7e77119da934290d254 Nov 02 20:20:57.628830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:57.628804 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" Nov 02 20:20:57.675639 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:20:57.675604 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" event={"ID":"3819be82-2b2e-4f1b-937d-5d90305375d0","Type":"ContainerStarted","Data":"80bbd5cf3009b2e21ef1e18dc5791a94a301e8d79a3ae7e77119da934290d254"} Nov 02 20:21:00.546290 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:00.546261 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:00.644230 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:00.644208 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wztv4" Nov 02 20:21:00.691377 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:00.691354 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" event={"ID":"3819be82-2b2e-4f1b-937d-5d90305375d0","Type":"ContainerStarted","Data":"2b92d5587790e9f54dd3ae87655f5eaa66a6814b29fb2c46596b088d7260d446"} Nov 02 20:21:00.691501 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:00.691401 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" Nov 02 20:21:00.788845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:00.788729 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:00.890182 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:00.890135 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" podStartSLOduration=1.678655038 podStartE2EDuration="4.890116792s" totalImagesPullingTime="3.211461754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:00.889601294 +0000 UTC m=+389.179257821" watchObservedRunningTime="2026-11-02 20:21:00.890116792 +0000 UTC m=+389.179773320" Nov 02 20:21:03.336931 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:21:03.336884 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:21:06.156475 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:06.156416 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:06.399320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:06.399283 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:08.748828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:08.748792 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:09.007857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:09.007734 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:11.698253 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:11.698172 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" Nov 02 20:21:12.139399 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:12.139369 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:12.402705 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:12.402583 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:13.241567 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:13.241519 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:13.487547 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:13.487518 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:14.533305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.533272 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hmvzj"] Nov 02 20:21:14.533736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.533518 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" podUID="3819be82-2b2e-4f1b-937d-5d90305375d0" containerName="limitador" containerID="cri-o://2b92d5587790e9f54dd3ae87655f5eaa66a6814b29fb2c46596b088d7260d446" gracePeriod=30 Nov 02 20:21:14.630853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.630826 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b"] Nov 02 20:21:14.681714 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.681678 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b"] Nov 02 20:21:14.681859 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.681811 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" Nov 02 20:21:14.733112 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.733055 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:21:14.733112 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.733105 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 02 20:21:14.733341 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.733110 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 20:21:14.733341 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.733275 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 02 20:21:14.733341 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.733325 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 02 20:21:14.742108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.742070 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-tkpmj\"" Nov 02 20:21:14.811692 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.811634 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/254dc9dc-96d4-4aec-99f2-249cadff0e49-manager-config\") pod \"jobset-controller-manager-577cc67499-k879b\" (UID: \"254dc9dc-96d4-4aec-99f2-249cadff0e49\") " pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" Nov 02 20:21:14.811692 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.811668 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254dc9dc-96d4-4aec-99f2-249cadff0e49-metrics-certs\") pod \"jobset-controller-manager-577cc67499-k879b\" (UID: \"254dc9dc-96d4-4aec-99f2-249cadff0e49\") " pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" Nov 02 20:21:14.811836 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.811706 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lmrt\" (UniqueName: \"kubernetes.io/projected/254dc9dc-96d4-4aec-99f2-249cadff0e49-kube-api-access-8lmrt\") pod \"jobset-controller-manager-577cc67499-k879b\" (UID: \"254dc9dc-96d4-4aec-99f2-249cadff0e49\") " pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" Nov 02 20:21:14.811836 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.811808 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/254dc9dc-96d4-4aec-99f2-249cadff0e49-cert\") pod \"jobset-controller-manager-577cc67499-k879b\" (UID: \"254dc9dc-96d4-4aec-99f2-249cadff0e49\") " pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" Nov 02 20:21:14.913046 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.913017 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/254dc9dc-96d4-4aec-99f2-249cadff0e49-manager-config\") pod \"jobset-controller-manager-577cc67499-k879b\" (UID: \"254dc9dc-96d4-4aec-99f2-249cadff0e49\") " pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" Nov 02 20:21:14.913169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.913055 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254dc9dc-96d4-4aec-99f2-249cadff0e49-metrics-certs\") pod \"jobset-controller-manager-577cc67499-k879b\" (UID: \"254dc9dc-96d4-4aec-99f2-249cadff0e49\") " pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" Nov 02 20:21:14.913169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.913116 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lmrt\" (UniqueName: \"kubernetes.io/projected/254dc9dc-96d4-4aec-99f2-249cadff0e49-kube-api-access-8lmrt\") pod \"jobset-controller-manager-577cc67499-k879b\" (UID: \"254dc9dc-96d4-4aec-99f2-249cadff0e49\") " pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" Nov 02 20:21:14.913169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.913145 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/254dc9dc-96d4-4aec-99f2-249cadff0e49-cert\") pod \"jobset-controller-manager-577cc67499-k879b\" (UID: \"254dc9dc-96d4-4aec-99f2-249cadff0e49\") " pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" Nov 02 20:21:14.913694 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.913671 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/254dc9dc-96d4-4aec-99f2-249cadff0e49-manager-config\") pod \"jobset-controller-manager-577cc67499-k879b\" (UID: \"254dc9dc-96d4-4aec-99f2-249cadff0e49\") " pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" Nov 02 20:21:14.915628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.915605 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/254dc9dc-96d4-4aec-99f2-249cadff0e49-cert\") pod \"jobset-controller-manager-577cc67499-k879b\" (UID: \"254dc9dc-96d4-4aec-99f2-249cadff0e49\") " pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" Nov 02 20:21:14.915920 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.915902 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/254dc9dc-96d4-4aec-99f2-249cadff0e49-metrics-certs\") pod \"jobset-controller-manager-577cc67499-k879b\" (UID: \"254dc9dc-96d4-4aec-99f2-249cadff0e49\") " pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" Nov 02 20:21:14.971437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:14.971409 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lmrt\" (UniqueName: \"kubernetes.io/projected/254dc9dc-96d4-4aec-99f2-249cadff0e49-kube-api-access-8lmrt\") pod \"jobset-controller-manager-577cc67499-k879b\" (UID: \"254dc9dc-96d4-4aec-99f2-249cadff0e49\") " pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" Nov 02 20:21:15.011038 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.011005 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" Nov 02 20:21:15.101562 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.101534 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" Nov 02 20:21:15.215819 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.215793 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3819be82-2b2e-4f1b-937d-5d90305375d0-kube-api-access-4mwtj\" (UniqueName: \"kubernetes.io/projected/3819be82-2b2e-4f1b-937d-5d90305375d0-kube-api-access-4mwtj\") pod \"3819be82-2b2e-4f1b-937d-5d90305375d0\" (UID: \"3819be82-2b2e-4f1b-937d-5d90305375d0\") " Nov 02 20:21:15.215958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.215907 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3819be82-2b2e-4f1b-937d-5d90305375d0-config-file\" (UniqueName: \"kubernetes.io/configmap/3819be82-2b2e-4f1b-937d-5d90305375d0-config-file\") pod \"3819be82-2b2e-4f1b-937d-5d90305375d0\" (UID: \"3819be82-2b2e-4f1b-937d-5d90305375d0\") " Nov 02 20:21:15.216267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.216243 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3819be82-2b2e-4f1b-937d-5d90305375d0-config-file" pod "3819be82-2b2e-4f1b-937d-5d90305375d0" (UID: "3819be82-2b2e-4f1b-937d-5d90305375d0"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:21:15.218279 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.218256 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3819be82-2b2e-4f1b-937d-5d90305375d0-kube-api-access-4mwtj" pod "3819be82-2b2e-4f1b-937d-5d90305375d0" (UID: "3819be82-2b2e-4f1b-937d-5d90305375d0"). InnerVolumeSpecName "kube-api-access-4mwtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:15.317436 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.317408 4971 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3819be82-2b2e-4f1b-937d-5d90305375d0-config-file\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:15.317436 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.317434 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mwtj\" (UniqueName: \"kubernetes.io/projected/3819be82-2b2e-4f1b-937d-5d90305375d0-kube-api-access-4mwtj\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:15.398444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.398422 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b"] Nov 02 20:21:15.399698 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:21:15.399676 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254dc9dc_96d4_4aec_99f2_249cadff0e49.slice/crio-117a12519060c305eda57560e62b0dd229e0c738acea9ec3158f36010e089ee5 WatchSource:0}: Error finding container 117a12519060c305eda57560e62b0dd229e0c738acea9ec3158f36010e089ee5: Status 404 returned error can't find the container with id 117a12519060c305eda57560e62b0dd229e0c738acea9ec3158f36010e089ee5 Nov 02 20:21:15.762687 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.762608 4971 generic.go:356] "Generic (PLEG): container finished" podID="3819be82-2b2e-4f1b-937d-5d90305375d0" containerID="2b92d5587790e9f54dd3ae87655f5eaa66a6814b29fb2c46596b088d7260d446" exitCode=0 Nov 02 20:21:15.762687 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.762654 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" Nov 02 20:21:15.763198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.762698 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" event={"ID":"3819be82-2b2e-4f1b-937d-5d90305375d0","Type":"ContainerDied","Data":"2b92d5587790e9f54dd3ae87655f5eaa66a6814b29fb2c46596b088d7260d446"} Nov 02 20:21:15.763198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.762755 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hmvzj" event={"ID":"3819be82-2b2e-4f1b-937d-5d90305375d0","Type":"ContainerDied","Data":"80bbd5cf3009b2e21ef1e18dc5791a94a301e8d79a3ae7e77119da934290d254"} Nov 02 20:21:15.763198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.762789 4971 scope.go:122] "RemoveContainer" containerID="2b92d5587790e9f54dd3ae87655f5eaa66a6814b29fb2c46596b088d7260d446" Nov 02 20:21:15.764157 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.764135 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" event={"ID":"254dc9dc-96d4-4aec-99f2-249cadff0e49","Type":"ContainerStarted","Data":"117a12519060c305eda57560e62b0dd229e0c738acea9ec3158f36010e089ee5"} Nov 02 20:21:15.771602 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.771586 4971 scope.go:122] "RemoveContainer" containerID="2b92d5587790e9f54dd3ae87655f5eaa66a6814b29fb2c46596b088d7260d446" Nov 02 20:21:15.771880 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:21:15.771859 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b92d5587790e9f54dd3ae87655f5eaa66a6814b29fb2c46596b088d7260d446\": container with ID starting with 2b92d5587790e9f54dd3ae87655f5eaa66a6814b29fb2c46596b088d7260d446 not found: ID does not exist" containerID="2b92d5587790e9f54dd3ae87655f5eaa66a6814b29fb2c46596b088d7260d446" Nov 02 20:21:15.771926 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.771885 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b92d5587790e9f54dd3ae87655f5eaa66a6814b29fb2c46596b088d7260d446"} err="failed to get container status \"2b92d5587790e9f54dd3ae87655f5eaa66a6814b29fb2c46596b088d7260d446\": rpc error: code = NotFound desc = could not find container \"2b92d5587790e9f54dd3ae87655f5eaa66a6814b29fb2c46596b088d7260d446\": container with ID starting with 2b92d5587790e9f54dd3ae87655f5eaa66a6814b29fb2c46596b088d7260d446 not found: ID does not exist" Nov 02 20:21:15.917265 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.917241 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hmvzj"] Nov 02 20:21:15.920645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:15.920627 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hmvzj"] Nov 02 20:21:17.551898 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:17.551867 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:17.803227 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:17.803158 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:17.864169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:17.864139 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3819be82-2b2e-4f1b-937d-5d90305375d0" path="/var/lib/kubelet/pods/3819be82-2b2e-4f1b-937d-5d90305375d0/volumes" Nov 02 20:21:18.601985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:18.601633 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq"] Nov 02 20:21:18.601985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:18.601837 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" podUID="84becb7d-d0d7-486c-82b1-36e4b1c4fe08" containerName="manager" containerID="cri-o://8de81e1c2b8feaccad6facd782cbfa6876eab287e231d56da85ff0815ad923bd" gracePeriod=10 Nov 02 20:21:18.616708 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:18.616660 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:18.783769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:18.783737 4971 generic.go:356] "Generic (PLEG): container finished" podID="84becb7d-d0d7-486c-82b1-36e4b1c4fe08" containerID="8de81e1c2b8feaccad6facd782cbfa6876eab287e231d56da85ff0815ad923bd" exitCode=0 Nov 02 20:21:18.783878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:18.783807 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" event={"ID":"84becb7d-d0d7-486c-82b1-36e4b1c4fe08","Type":"ContainerDied","Data":"8de81e1c2b8feaccad6facd782cbfa6876eab287e231d56da85ff0815ad923bd"} Nov 02 20:21:18.862550 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:18.862490 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:18.866411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:18.866386 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" Nov 02 20:21:18.947061 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:18.947035 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84becb7d-d0d7-486c-82b1-36e4b1c4fe08-kube-api-access-6zrfj\" (UniqueName: \"kubernetes.io/projected/84becb7d-d0d7-486c-82b1-36e4b1c4fe08-kube-api-access-6zrfj\") pod \"84becb7d-d0d7-486c-82b1-36e4b1c4fe08\" (UID: \"84becb7d-d0d7-486c-82b1-36e4b1c4fe08\") " Nov 02 20:21:18.947218 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:18.947106 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84becb7d-d0d7-486c-82b1-36e4b1c4fe08-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84becb7d-d0d7-486c-82b1-36e4b1c4fe08-extensions-socket-volume\") pod \"84becb7d-d0d7-486c-82b1-36e4b1c4fe08\" (UID: \"84becb7d-d0d7-486c-82b1-36e4b1c4fe08\") " Nov 02 20:21:18.947575 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:18.947547 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84becb7d-d0d7-486c-82b1-36e4b1c4fe08-extensions-socket-volume" pod "84becb7d-d0d7-486c-82b1-36e4b1c4fe08" (UID: "84becb7d-d0d7-486c-82b1-36e4b1c4fe08"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:18.949389 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:18.949364 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84becb7d-d0d7-486c-82b1-36e4b1c4fe08-kube-api-access-6zrfj" pod "84becb7d-d0d7-486c-82b1-36e4b1c4fe08" (UID: "84becb7d-d0d7-486c-82b1-36e4b1c4fe08"). InnerVolumeSpecName "kube-api-access-6zrfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:19.048195 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:19.048166 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zrfj\" (UniqueName: \"kubernetes.io/projected/84becb7d-d0d7-486c-82b1-36e4b1c4fe08-kube-api-access-6zrfj\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:19.048195 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:19.048192 4971 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84becb7d-d0d7-486c-82b1-36e4b1c4fe08-extensions-socket-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:19.150072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:19.149925 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:19.424667 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:19.424551 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:19.791640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:19.791561 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" event={"ID":"84becb7d-d0d7-486c-82b1-36e4b1c4fe08","Type":"ContainerDied","Data":"695020f7a5a418825c6a5d4ff2317cbbd40cefb5457a8199042675c2cf3adf00"} Nov 02 20:21:19.791640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:19.791605 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq" Nov 02 20:21:19.792215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:19.791668 4971 scope.go:122] "RemoveContainer" containerID="8de81e1c2b8feaccad6facd782cbfa6876eab287e231d56da85ff0815ad923bd" Nov 02 20:21:19.993351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:19.993314 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq"] Nov 02 20:21:20.040272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:20.040250 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m65rq"] Nov 02 20:21:20.799151 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:20.799113 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" event={"ID":"254dc9dc-96d4-4aec-99f2-249cadff0e49","Type":"ContainerStarted","Data":"660970751b9b2f09198153923034d2602709a536360f9a8bf17c79f3627c518f"} Nov 02 20:21:20.799561 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:20.799215 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" Nov 02 20:21:20.910068 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:20.910023 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" podStartSLOduration=2.312088621 podStartE2EDuration="6.91000951s" totalImagesPullingTime="4.597920889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:20.908693961 +0000 UTC m=+409.198350488" watchObservedRunningTime="2026-11-02 20:21:20.91000951 +0000 UTC m=+409.199666037" Nov 02 20:21:21.864437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:21.864400 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84becb7d-d0d7-486c-82b1-36e4b1c4fe08" path="/var/lib/kubelet/pods/84becb7d-d0d7-486c-82b1-36e4b1c4fe08/volumes" Nov 02 20:21:23.884803 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:23.884770 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:25.159794 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:25.159752 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:25.429922 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:25.429825 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:27.762102 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:27.762059 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:28.004525 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:28.004491 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:31.362888 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:31.362844 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:31.614433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:31.614345 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:31.810017 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:31.809981 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-577cc67499-k879b" Nov 02 20:21:31.861346 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:31.861306 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:32.146319 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:32.146283 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:33.338002 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:21:33.337965 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:21:38.024301 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:38.024264 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:38.266054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:38.266024 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:39.351419 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:39.351377 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fh9lt"] Nov 02 20:21:39.351895 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:39.351785 4971 cpu_manager.go:414] "containerMap: removing container" podUID="3819be82-2b2e-4f1b-937d-5d90305375d0" containerName="limitador" Nov 02 20:21:39.351895 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:39.351798 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3819be82-2b2e-4f1b-937d-5d90305375d0" containerName="limitador" Nov 02 20:21:39.351895 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:39.351810 4971 cpu_manager.go:414] "containerMap: removing container" podUID="84becb7d-d0d7-486c-82b1-36e4b1c4fe08" containerName="manager" Nov 02 20:21:39.351895 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:39.351815 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84becb7d-d0d7-486c-82b1-36e4b1c4fe08" containerName="manager" Nov 02 20:21:39.352023 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:39.351902 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="84becb7d-d0d7-486c-82b1-36e4b1c4fe08" containerName="manager" Nov 02 20:21:39.352023 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:39.351917 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="3819be82-2b2e-4f1b-937d-5d90305375d0" containerName="limitador" Nov 02 20:21:39.355685 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:39.355664 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fh9lt" Nov 02 20:21:39.405899 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:39.405874 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gvxgg\"" Nov 02 20:21:39.407542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:39.407517 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fh9lt"] Nov 02 20:21:39.507007 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:39.506979 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxrzg\" (UniqueName: \"kubernetes.io/projected/913d5c7f-6064-4262-b624-b9f65afb7c66-kube-api-access-rxrzg\") pod \"authorino-56c6c9ddd5-fh9lt\" (UID: \"913d5c7f-6064-4262-b624-b9f65afb7c66\") " pod="kuadrant-system/authorino-56c6c9ddd5-fh9lt" Nov 02 20:21:39.608260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:39.608194 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxrzg\" (UniqueName: \"kubernetes.io/projected/913d5c7f-6064-4262-b624-b9f65afb7c66-kube-api-access-rxrzg\") pod \"authorino-56c6c9ddd5-fh9lt\" (UID: \"913d5c7f-6064-4262-b624-b9f65afb7c66\") " pod="kuadrant-system/authorino-56c6c9ddd5-fh9lt" Nov 02 20:21:39.666194 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:39.666164 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxrzg\" (UniqueName: \"kubernetes.io/projected/913d5c7f-6064-4262-b624-b9f65afb7c66-kube-api-access-rxrzg\") pod \"authorino-56c6c9ddd5-fh9lt\" (UID: \"913d5c7f-6064-4262-b624-b9f65afb7c66\") " pod="kuadrant-system/authorino-56c6c9ddd5-fh9lt" Nov 02 20:21:39.680790 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:39.680768 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fh9lt" Nov 02 20:21:40.046479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:40.046441 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fh9lt"] Nov 02 20:21:40.048699 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:21:40.048668 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod913d5c7f_6064_4262_b624_b9f65afb7c66.slice/crio-cfd6431e7fe92df60c36907fbd2415f26a0aa1bb5f13b3c0440cce85dcb035d3 WatchSource:0}: Error finding container cfd6431e7fe92df60c36907fbd2415f26a0aa1bb5f13b3c0440cce85dcb035d3: Status 404 returned error can't find the container with id cfd6431e7fe92df60c36907fbd2415f26a0aa1bb5f13b3c0440cce85dcb035d3 Nov 02 20:21:40.893407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:40.893370 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fh9lt" event={"ID":"913d5c7f-6064-4262-b624-b9f65afb7c66","Type":"ContainerStarted","Data":"cfd6431e7fe92df60c36907fbd2415f26a0aa1bb5f13b3c0440cce85dcb035d3"} Nov 02 20:21:44.601665 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:44.601629 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:44.865064 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:44.864983 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:45.334112 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:45.334015 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rqdnx"] Nov 02 20:21:45.342905 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:45.342865 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqdnx" Nov 02 20:21:45.390724 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:45.390692 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rqdnx"] Nov 02 20:21:45.435627 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:45.435599 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:21:45.435627 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:45.435620 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 20:21:45.435820 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:45.435801 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:21:45.435880 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:45.435845 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-n8mn2\"" Nov 02 20:21:45.458778 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:45.458754 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rkg\" (UniqueName: \"kubernetes.io/projected/57bf0b41-d2ed-4d88-9ecf-8d46a2b2639c-kube-api-access-w2rkg\") pod \"postgres-7cdbf6dc44-rqdnx\" (UID: \"57bf0b41-d2ed-4d88-9ecf-8d46a2b2639c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqdnx" Nov 02 20:21:45.458945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:45.458875 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/57bf0b41-d2ed-4d88-9ecf-8d46a2b2639c-data\") pod \"postgres-7cdbf6dc44-rqdnx\" (UID: \"57bf0b41-d2ed-4d88-9ecf-8d46a2b2639c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqdnx" Nov 02 20:21:45.533597 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:45.533562 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:45.559918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:45.559886 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rkg\" (UniqueName: \"kubernetes.io/projected/57bf0b41-d2ed-4d88-9ecf-8d46a2b2639c-kube-api-access-w2rkg\") pod \"postgres-7cdbf6dc44-rqdnx\" (UID: \"57bf0b41-d2ed-4d88-9ecf-8d46a2b2639c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqdnx" Nov 02 20:21:45.560057 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:45.559966 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/57bf0b41-d2ed-4d88-9ecf-8d46a2b2639c-data\") pod \"postgres-7cdbf6dc44-rqdnx\" (UID: \"57bf0b41-d2ed-4d88-9ecf-8d46a2b2639c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqdnx" Nov 02 20:21:45.560378 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:45.560360 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/57bf0b41-d2ed-4d88-9ecf-8d46a2b2639c-data\") pod \"postgres-7cdbf6dc44-rqdnx\" (UID: \"57bf0b41-d2ed-4d88-9ecf-8d46a2b2639c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqdnx" Nov 02 20:21:45.618276 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:45.618253 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rkg\" (UniqueName: \"kubernetes.io/projected/57bf0b41-d2ed-4d88-9ecf-8d46a2b2639c-kube-api-access-w2rkg\") pod \"postgres-7cdbf6dc44-rqdnx\" (UID: \"57bf0b41-d2ed-4d88-9ecf-8d46a2b2639c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqdnx" Nov 02 20:21:45.659112 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:45.659072 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqdnx" Nov 02 20:21:46.031962 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:46.031935 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rqdnx"] Nov 02 20:21:46.034585 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:21:46.034560 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57bf0b41_d2ed_4d88_9ecf_8d46a2b2639c.slice/crio-c61479d22e83415dae5e065987e74546128699a72f6fc8c0bba56f0ced47fbd4 WatchSource:0}: Error finding container c61479d22e83415dae5e065987e74546128699a72f6fc8c0bba56f0ced47fbd4: Status 404 returned error can't find the container with id c61479d22e83415dae5e065987e74546128699a72f6fc8c0bba56f0ced47fbd4 Nov 02 20:21:46.922267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:46.922231 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fh9lt" event={"ID":"913d5c7f-6064-4262-b624-b9f65afb7c66","Type":"ContainerStarted","Data":"8cfc317abacc2cfa0a0e47cad54d1f730e60f07be5c38d1f3d6ece2a3ae93667"} Nov 02 20:21:46.924901 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:46.924868 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqdnx" event={"ID":"57bf0b41-d2ed-4d88-9ecf-8d46a2b2639c","Type":"ContainerStarted","Data":"c61479d22e83415dae5e065987e74546128699a72f6fc8c0bba56f0ced47fbd4"} Nov 02 20:21:47.030141 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:47.030072 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-fh9lt" podStartSLOduration=1.731375898 podStartE2EDuration="8.030059086s" totalImagesPullingTime="6.298683188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:47.028913536 +0000 UTC m=+435.318570064" watchObservedRunningTime="2026-11-02 20:21:47.030059086 +0000 UTC m=+435.319715633" Nov 02 20:21:47.765118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:47.765061 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:48.012145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:48.012103 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:21:55.968144 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:55.968111 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqdnx" event={"ID":"57bf0b41-d2ed-4d88-9ecf-8d46a2b2639c","Type":"ContainerStarted","Data":"4d44f774656b91a8c961563d22b78e0060c1b9248b810fb5ff08d8bffd76648f"} Nov 02 20:21:55.968567 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:55.968280 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqdnx" Nov 02 20:21:56.078164 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:21:56.078102 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqdnx" podStartSLOduration=2.055492525 podStartE2EDuration="11.078065372s" totalImagesPullingTime="9.022572847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:56.076670541 +0000 UTC m=+444.366327068" watchObservedRunningTime="2026-11-02 20:21:56.078065372 +0000 UTC m=+444.367721899" Nov 02 20:22:02.009741 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:02.009714 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rqdnx" Nov 02 20:22:03.336563 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:22:03.336522 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:22:09.226886 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:09.226847 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:09.474738 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:09.474701 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:09.725745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:09.725719 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:13.836779 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:13.836717 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:14.080420 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:14.080385 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:16.760483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:16.760449 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:17.023069 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:17.022990 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:24.243164 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:24.243129 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:24.488610 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:24.488580 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:33.339255 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:22:33.339214 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:22:44.686652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:44.686563 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:44.987945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:44.987856 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:46.721468 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:46.721427 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:46.985227 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:46.985149 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:47.234264 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:47.234236 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:49.771164 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:49.771125 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:50.025642 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.025525 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:50.277939 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.277861 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:50.529868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.529786 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:50.582230 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.582192 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-8ggqj"] Nov 02 20:22:50.587114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.587070 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-8ggqj" Nov 02 20:22:50.637070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.637043 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-8ggqj"] Nov 02 20:22:50.686471 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.686444 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hpj2\" (UniqueName: \"kubernetes.io/projected/b2c5de9e-e657-4c2b-a5bc-7807bf957e61-kube-api-access-2hpj2\") pod \"kserve-controller-manager-776cc66d47-8ggqj\" (UID: \"b2c5de9e-e657-4c2b-a5bc-7807bf957e61\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8ggqj" Nov 02 20:22:50.686599 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.686512 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2c5de9e-e657-4c2b-a5bc-7807bf957e61-cert\") pod \"kserve-controller-manager-776cc66d47-8ggqj\" (UID: \"b2c5de9e-e657-4c2b-a5bc-7807bf957e61\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8ggqj" Nov 02 20:22:50.686851 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.686826 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-zwgnf\"" Nov 02 20:22:50.687001 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.686868 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 02 20:22:50.687001 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.686939 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 20:22:50.687140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.686943 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 20:22:50.783719 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.783654 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:50.787781 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.787758 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2c5de9e-e657-4c2b-a5bc-7807bf957e61-cert\") pod \"kserve-controller-manager-776cc66d47-8ggqj\" (UID: \"b2c5de9e-e657-4c2b-a5bc-7807bf957e61\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8ggqj" Nov 02 20:22:50.787892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.787823 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hpj2\" (UniqueName: \"kubernetes.io/projected/b2c5de9e-e657-4c2b-a5bc-7807bf957e61-kube-api-access-2hpj2\") pod \"kserve-controller-manager-776cc66d47-8ggqj\" (UID: \"b2c5de9e-e657-4c2b-a5bc-7807bf957e61\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8ggqj" Nov 02 20:22:50.790249 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.790215 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2c5de9e-e657-4c2b-a5bc-7807bf957e61-cert\") pod \"kserve-controller-manager-776cc66d47-8ggqj\" (UID: \"b2c5de9e-e657-4c2b-a5bc-7807bf957e61\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8ggqj" Nov 02 20:22:50.847907 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.847886 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hpj2\" (UniqueName: \"kubernetes.io/projected/b2c5de9e-e657-4c2b-a5bc-7807bf957e61-kube-api-access-2hpj2\") pod \"kserve-controller-manager-776cc66d47-8ggqj\" (UID: \"b2c5de9e-e657-4c2b-a5bc-7807bf957e61\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8ggqj" Nov 02 20:22:50.904076 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.904057 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-8ggqj" Nov 02 20:22:50.924862 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.924826 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-dd6xp"] Nov 02 20:22:50.928185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.928163 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-dd6xp" Nov 02 20:22:50.982027 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:50.981994 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-dd6xp"] Nov 02 20:22:51.022730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:51.022706 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wg6rk\"" Nov 02 20:22:51.022730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:51.022707 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 20:22:51.022730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:51.022707 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 20:22:51.031156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:51.031108 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:51.091203 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:51.091150 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f80e1293-d112-436c-bb77-31eea6da6fe2-cert\") pod \"odh-model-controller-769875db56-dd6xp\" (UID: \"f80e1293-d112-436c-bb77-31eea6da6fe2\") " pod="opendatahub/odh-model-controller-769875db56-dd6xp" Nov 02 20:22:51.091203 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:51.091183 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f80e1293-d112-436c-bb77-31eea6da6fe2-metrics-certs\") pod \"odh-model-controller-769875db56-dd6xp\" (UID: \"f80e1293-d112-436c-bb77-31eea6da6fe2\") " pod="opendatahub/odh-model-controller-769875db56-dd6xp" Nov 02 20:22:51.091338 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:51.091208 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j6l6\" (UniqueName: \"kubernetes.io/projected/f80e1293-d112-436c-bb77-31eea6da6fe2-kube-api-access-7j6l6\") pod \"odh-model-controller-769875db56-dd6xp\" (UID: \"f80e1293-d112-436c-bb77-31eea6da6fe2\") " pod="opendatahub/odh-model-controller-769875db56-dd6xp" Nov 02 20:22:51.191757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:51.191733 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f80e1293-d112-436c-bb77-31eea6da6fe2-cert\") pod \"odh-model-controller-769875db56-dd6xp\" (UID: \"f80e1293-d112-436c-bb77-31eea6da6fe2\") " pod="opendatahub/odh-model-controller-769875db56-dd6xp" Nov 02 20:22:51.191870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:51.191766 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f80e1293-d112-436c-bb77-31eea6da6fe2-metrics-certs\") pod \"odh-model-controller-769875db56-dd6xp\" (UID: \"f80e1293-d112-436c-bb77-31eea6da6fe2\") " pod="opendatahub/odh-model-controller-769875db56-dd6xp" Nov 02 20:22:51.191870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:51.191793 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j6l6\" (UniqueName: \"kubernetes.io/projected/f80e1293-d112-436c-bb77-31eea6da6fe2-kube-api-access-7j6l6\") pod \"odh-model-controller-769875db56-dd6xp\" (UID: \"f80e1293-d112-436c-bb77-31eea6da6fe2\") " pod="opendatahub/odh-model-controller-769875db56-dd6xp" Nov 02 20:22:51.194410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:51.194388 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f80e1293-d112-436c-bb77-31eea6da6fe2-metrics-certs\") pod \"odh-model-controller-769875db56-dd6xp\" (UID: \"f80e1293-d112-436c-bb77-31eea6da6fe2\") " pod="opendatahub/odh-model-controller-769875db56-dd6xp" Nov 02 20:22:51.194511 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:51.194479 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f80e1293-d112-436c-bb77-31eea6da6fe2-cert\") pod \"odh-model-controller-769875db56-dd6xp\" (UID: \"f80e1293-d112-436c-bb77-31eea6da6fe2\") " pod="opendatahub/odh-model-controller-769875db56-dd6xp" Nov 02 20:22:51.249662 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:51.249635 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j6l6\" (UniqueName: \"kubernetes.io/projected/f80e1293-d112-436c-bb77-31eea6da6fe2-kube-api-access-7j6l6\") pod \"odh-model-controller-769875db56-dd6xp\" (UID: \"f80e1293-d112-436c-bb77-31eea6da6fe2\") " pod="opendatahub/odh-model-controller-769875db56-dd6xp" Nov 02 20:22:51.268434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:51.268411 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-8ggqj"] Nov 02 20:22:51.270027 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:22:51.270002 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c5de9e_e657_4c2b_a5bc_7807bf957e61.slice/crio-8446ad0a919030c976f3ed18773f03b942277d7b7179b8d800abe8d4912c38bb WatchSource:0}: Error finding container 8446ad0a919030c976f3ed18773f03b942277d7b7179b8d800abe8d4912c38bb: Status 404 returned error can't find the container with id 8446ad0a919030c976f3ed18773f03b942277d7b7179b8d800abe8d4912c38bb Nov 02 20:22:51.540950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:51.540924 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-dd6xp" Nov 02 20:22:51.903850 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:51.903826 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-dd6xp"] Nov 02 20:22:51.905860 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:22:51.905832 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80e1293_d112_436c_bb77_31eea6da6fe2.slice/crio-e78ba7eb1b21773e19c70ac126919e4c10f6ca7711cf632f57c4bc2941288b6a WatchSource:0}: Error finding container e78ba7eb1b21773e19c70ac126919e4c10f6ca7711cf632f57c4bc2941288b6a: Status 404 returned error can't find the container with id e78ba7eb1b21773e19c70ac126919e4c10f6ca7711cf632f57c4bc2941288b6a Nov 02 20:22:52.111277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:52.111245 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:52.245758 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:52.245671 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-8ggqj" event={"ID":"b2c5de9e-e657-4c2b-a5bc-7807bf957e61","Type":"ContainerStarted","Data":"8446ad0a919030c976f3ed18773f03b942277d7b7179b8d800abe8d4912c38bb"} Nov 02 20:22:52.247181 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:52.247151 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-dd6xp" event={"ID":"f80e1293-d112-436c-bb77-31eea6da6fe2","Type":"ContainerStarted","Data":"e78ba7eb1b21773e19c70ac126919e4c10f6ca7711cf632f57c4bc2941288b6a"} Nov 02 20:22:54.997983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:54.997884 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:55.247128 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:55.247076 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:55.266885 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:55.266804 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-8ggqj" event={"ID":"b2c5de9e-e657-4c2b-a5bc-7807bf957e61","Type":"ContainerStarted","Data":"11990d78601465b3a370e3255169fbedfb6e601c8c42644d2be97d44d2baf944"} Nov 02 20:22:55.267044 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:55.266947 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-8ggqj" Nov 02 20:22:55.378152 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:55.378061 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-8ggqj" podStartSLOduration=1.912583287 podStartE2EDuration="5.378043707s" totalImagesPullingTime="3.46546042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:55.376813754 +0000 UTC m=+503.666470282" watchObservedRunningTime="2026-11-02 20:22:55.378043707 +0000 UTC m=+503.667700236" Nov 02 20:22:56.273558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:56.273526 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-dd6xp" event={"ID":"f80e1293-d112-436c-bb77-31eea6da6fe2","Type":"ContainerStarted","Data":"1e693d64bbb9a0b4f84a8090a3b1770782676328aa03ff0e302b6e89802abc3b"} Nov 02 20:22:56.273939 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:56.273619 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-dd6xp" Nov 02 20:22:56.383370 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:56.383324 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-dd6xp" podStartSLOduration=2.917648527 podStartE2EDuration="6.383311468s" totalImagesPullingTime="3.465662941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:56.382704769 +0000 UTC m=+504.672361300" watchObservedRunningTime="2026-11-02 20:22:56.383311468 +0000 UTC m=+504.672967995" Nov 02 20:22:57.371415 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:57.371362 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:57.611468 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:57.611435 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:57.860035 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:57.860006 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:58.104233 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:58.104206 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:58.354486 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:58.354456 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:22:58.665110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:22:58.665032 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:03.336897 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:23:03.336862 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:23:07.280588 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:07.280561 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-dd6xp" Nov 02 20:23:07.745826 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:07.745788 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:07.904107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:07.904055 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-56rqk"] Nov 02 20:23:07.907840 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:07.907817 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-56rqk" Nov 02 20:23:07.936843 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:07.936820 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d9fs\" (UniqueName: \"kubernetes.io/projected/973c7ab3-8cc7-4df1-8a7b-70a2288dd280-kube-api-access-6d9fs\") pod \"authorino-6f6944bcdd-56rqk\" (UID: \"973c7ab3-8cc7-4df1-8a7b-70a2288dd280\") " pod="kuadrant-system/authorino-6f6944bcdd-56rqk" Nov 02 20:23:07.960647 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:07.960626 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-56rqk"] Nov 02 20:23:08.016248 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:08.016171 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:08.038205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:08.038180 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d9fs\" (UniqueName: \"kubernetes.io/projected/973c7ab3-8cc7-4df1-8a7b-70a2288dd280-kube-api-access-6d9fs\") pod \"authorino-6f6944bcdd-56rqk\" (UID: \"973c7ab3-8cc7-4df1-8a7b-70a2288dd280\") " pod="kuadrant-system/authorino-6f6944bcdd-56rqk" Nov 02 20:23:08.096844 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:08.096822 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d9fs\" (UniqueName: \"kubernetes.io/projected/973c7ab3-8cc7-4df1-8a7b-70a2288dd280-kube-api-access-6d9fs\") pod \"authorino-6f6944bcdd-56rqk\" (UID: \"973c7ab3-8cc7-4df1-8a7b-70a2288dd280\") " pod="kuadrant-system/authorino-6f6944bcdd-56rqk" Nov 02 20:23:08.220337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:08.220308 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-56rqk" Nov 02 20:23:08.266001 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:08.265969 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-56rqk"] Nov 02 20:23:08.291707 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:08.291679 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75f848775b-57bcc"] Nov 02 20:23:08.299545 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:08.299525 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75f848775b-57bcc" Nov 02 20:23:08.340777 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:08.340751 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-886dk\" (UniqueName: \"kubernetes.io/projected/d6d75d87-17ea-452c-9f1e-b2554ed5b6c1-kube-api-access-886dk\") pod \"authorino-75f848775b-57bcc\" (UID: \"d6d75d87-17ea-452c-9f1e-b2554ed5b6c1\") " pod="kuadrant-system/authorino-75f848775b-57bcc" Nov 02 20:23:08.344819 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:08.344794 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75f848775b-57bcc"] Nov 02 20:23:08.441780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:08.441756 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-886dk\" (UniqueName: \"kubernetes.io/projected/d6d75d87-17ea-452c-9f1e-b2554ed5b6c1-kube-api-access-886dk\") pod \"authorino-75f848775b-57bcc\" (UID: \"d6d75d87-17ea-452c-9f1e-b2554ed5b6c1\") " pod="kuadrant-system/authorino-75f848775b-57bcc" Nov 02 20:23:08.500633 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:08.500609 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-886dk\" (UniqueName: \"kubernetes.io/projected/d6d75d87-17ea-452c-9f1e-b2554ed5b6c1-kube-api-access-886dk\") pod \"authorino-75f848775b-57bcc\" (UID: \"d6d75d87-17ea-452c-9f1e-b2554ed5b6c1\") " pod="kuadrant-system/authorino-75f848775b-57bcc" Nov 02 20:23:08.581751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:08.581731 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-56rqk"] Nov 02 20:23:08.583644 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:23:08.583619 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod973c7ab3_8cc7_4df1_8a7b_70a2288dd280.slice/crio-c32d91ebbc2226c9a43cd22ffa3fc222baa5a1f040d9925af848d170d124becb WatchSource:0}: Error finding container c32d91ebbc2226c9a43cd22ffa3fc222baa5a1f040d9925af848d170d124becb: Status 404 returned error can't find the container with id c32d91ebbc2226c9a43cd22ffa3fc222baa5a1f040d9925af848d170d124becb Nov 02 20:23:08.614801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:08.614777 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75f848775b-57bcc" Nov 02 20:23:08.621751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:08.621729 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75f848775b-57bcc"] Nov 02 20:23:08.979564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:08.979543 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75f848775b-57bcc"] Nov 02 20:23:08.981251 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:23:08.981226 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6d75d87_17ea_452c_9f1e_b2554ed5b6c1.slice/crio-6c8d85894203eb622d70b726631ad2935d55fb1b479fccce472e6ef3c1d8d4d5 WatchSource:0}: Error finding container 6c8d85894203eb622d70b726631ad2935d55fb1b479fccce472e6ef3c1d8d4d5: Status 404 returned error can't find the container with id 6c8d85894203eb622d70b726631ad2935d55fb1b479fccce472e6ef3c1d8d4d5 Nov 02 20:23:09.340332 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:09.340253 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-56rqk" event={"ID":"973c7ab3-8cc7-4df1-8a7b-70a2288dd280","Type":"ContainerStarted","Data":"c32d91ebbc2226c9a43cd22ffa3fc222baa5a1f040d9925af848d170d124becb"} Nov 02 20:23:09.341514 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:09.341483 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75f848775b-57bcc" event={"ID":"d6d75d87-17ea-452c-9f1e-b2554ed5b6c1","Type":"ContainerStarted","Data":"6c8d85894203eb622d70b726631ad2935d55fb1b479fccce472e6ef3c1d8d4d5"} Nov 02 20:23:11.354952 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:11.354908 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-56rqk" event={"ID":"973c7ab3-8cc7-4df1-8a7b-70a2288dd280","Type":"ContainerStarted","Data":"1bc8538b7f0e5ca9cdb55985d07ca6f318a0bc5c31eaec131a3837c01795bea8"} Nov 02 20:23:11.355411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:11.355094 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-56rqk" podUID="973c7ab3-8cc7-4df1-8a7b-70a2288dd280" containerName="authorino" containerID="cri-o://1bc8538b7f0e5ca9cdb55985d07ca6f318a0bc5c31eaec131a3837c01795bea8" gracePeriod=30 Nov 02 20:23:11.357982 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:11.357953 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75f848775b-57bcc" event={"ID":"d6d75d87-17ea-452c-9f1e-b2554ed5b6c1","Type":"ContainerStarted","Data":"76aea97ad079382377056170bc1fc7ab282b5cdddb820ec81895da0811fed366"} Nov 02 20:23:11.358163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:11.357996 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-75f848775b-57bcc" podUID="d6d75d87-17ea-452c-9f1e-b2554ed5b6c1" containerName="authorino" containerID="cri-o://76aea97ad079382377056170bc1fc7ab282b5cdddb820ec81895da0811fed366" gracePeriod=30 Nov 02 20:23:11.468203 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:11.468139 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-56rqk" podStartSLOduration=2.63670985 podStartE2EDuration="4.468120352s" totalImagesPullingTime="1.831410502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:11.466115419 +0000 UTC m=+519.755771948" watchObservedRunningTime="2026-11-02 20:23:11.468120352 +0000 UTC m=+519.757776881" Nov 02 20:23:11.580786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:11.580737 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-75f848775b-57bcc" podStartSLOduration=2.024087237 podStartE2EDuration="3.580721994s" totalImagesPullingTime="1.556634757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:11.579928514 +0000 UTC m=+519.869585045" watchObservedRunningTime="2026-11-02 20:23:11.580721994 +0000 UTC m=+519.870378522" Nov 02 20:23:11.641696 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:11.641672 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-56rqk" Nov 02 20:23:11.644634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:11.644614 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75f848775b-57bcc" Nov 02 20:23:11.771557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:11.771458 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/973c7ab3-8cc7-4df1-8a7b-70a2288dd280-kube-api-access-6d9fs\" (UniqueName: \"kubernetes.io/projected/973c7ab3-8cc7-4df1-8a7b-70a2288dd280-kube-api-access-6d9fs\") pod \"973c7ab3-8cc7-4df1-8a7b-70a2288dd280\" (UID: \"973c7ab3-8cc7-4df1-8a7b-70a2288dd280\") " Nov 02 20:23:11.771750 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:11.771559 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d75d87-17ea-452c-9f1e-b2554ed5b6c1-kube-api-access-886dk\" (UniqueName: \"kubernetes.io/projected/d6d75d87-17ea-452c-9f1e-b2554ed5b6c1-kube-api-access-886dk\") pod \"d6d75d87-17ea-452c-9f1e-b2554ed5b6c1\" (UID: \"d6d75d87-17ea-452c-9f1e-b2554ed5b6c1\") " Nov 02 20:23:11.774076 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:11.774042 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d75d87-17ea-452c-9f1e-b2554ed5b6c1-kube-api-access-886dk" pod "d6d75d87-17ea-452c-9f1e-b2554ed5b6c1" (UID: "d6d75d87-17ea-452c-9f1e-b2554ed5b6c1"). InnerVolumeSpecName "kube-api-access-886dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:11.774344 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:11.774322 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/973c7ab3-8cc7-4df1-8a7b-70a2288dd280-kube-api-access-6d9fs" pod "973c7ab3-8cc7-4df1-8a7b-70a2288dd280" (UID: "973c7ab3-8cc7-4df1-8a7b-70a2288dd280"). InnerVolumeSpecName "kube-api-access-6d9fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:11.875373 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:11.875300 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-886dk\" (UniqueName: \"kubernetes.io/projected/d6d75d87-17ea-452c-9f1e-b2554ed5b6c1-kube-api-access-886dk\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:11.875373 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:11.875336 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d9fs\" (UniqueName: \"kubernetes.io/projected/973c7ab3-8cc7-4df1-8a7b-70a2288dd280-kube-api-access-6d9fs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:12.091842 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.091809 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fh9lt"] Nov 02 20:23:12.092014 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.091990 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-fh9lt" podUID="913d5c7f-6064-4262-b624-b9f65afb7c66" containerName="authorino" containerID="cri-o://8cfc317abacc2cfa0a0e47cad54d1f730e60f07be5c38d1f3d6ece2a3ae93667" gracePeriod=30 Nov 02 20:23:12.330822 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.330796 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fh9lt" Nov 02 20:23:12.364288 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.364262 4971 generic.go:356] "Generic (PLEG): container finished" podID="913d5c7f-6064-4262-b624-b9f65afb7c66" containerID="8cfc317abacc2cfa0a0e47cad54d1f730e60f07be5c38d1f3d6ece2a3ae93667" exitCode=0 Nov 02 20:23:12.364660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.364299 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fh9lt" Nov 02 20:23:12.364660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.364348 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fh9lt" event={"ID":"913d5c7f-6064-4262-b624-b9f65afb7c66","Type":"ContainerDied","Data":"8cfc317abacc2cfa0a0e47cad54d1f730e60f07be5c38d1f3d6ece2a3ae93667"} Nov 02 20:23:12.364660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.364393 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fh9lt" event={"ID":"913d5c7f-6064-4262-b624-b9f65afb7c66","Type":"ContainerDied","Data":"cfd6431e7fe92df60c36907fbd2415f26a0aa1bb5f13b3c0440cce85dcb035d3"} Nov 02 20:23:12.364660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.364410 4971 scope.go:122] "RemoveContainer" containerID="8cfc317abacc2cfa0a0e47cad54d1f730e60f07be5c38d1f3d6ece2a3ae93667" Nov 02 20:23:12.365598 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.365574 4971 generic.go:356] "Generic (PLEG): container finished" podID="973c7ab3-8cc7-4df1-8a7b-70a2288dd280" containerID="1bc8538b7f0e5ca9cdb55985d07ca6f318a0bc5c31eaec131a3837c01795bea8" exitCode=0 Nov 02 20:23:12.365706 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.365616 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-56rqk" Nov 02 20:23:12.365758 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.365663 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-56rqk" event={"ID":"973c7ab3-8cc7-4df1-8a7b-70a2288dd280","Type":"ContainerDied","Data":"1bc8538b7f0e5ca9cdb55985d07ca6f318a0bc5c31eaec131a3837c01795bea8"} Nov 02 20:23:12.365875 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.365859 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-56rqk" event={"ID":"973c7ab3-8cc7-4df1-8a7b-70a2288dd280","Type":"ContainerDied","Data":"c32d91ebbc2226c9a43cd22ffa3fc222baa5a1f040d9925af848d170d124becb"} Nov 02 20:23:12.367051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.367023 4971 generic.go:356] "Generic (PLEG): container finished" podID="d6d75d87-17ea-452c-9f1e-b2554ed5b6c1" containerID="76aea97ad079382377056170bc1fc7ab282b5cdddb820ec81895da0811fed366" exitCode=0 Nov 02 20:23:12.367161 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.367059 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75f848775b-57bcc" event={"ID":"d6d75d87-17ea-452c-9f1e-b2554ed5b6c1","Type":"ContainerDied","Data":"76aea97ad079382377056170bc1fc7ab282b5cdddb820ec81895da0811fed366"} Nov 02 20:23:12.367161 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.367070 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75f848775b-57bcc" Nov 02 20:23:12.367161 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.367101 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75f848775b-57bcc" event={"ID":"d6d75d87-17ea-452c-9f1e-b2554ed5b6c1","Type":"ContainerDied","Data":"6c8d85894203eb622d70b726631ad2935d55fb1b479fccce472e6ef3c1d8d4d5"} Nov 02 20:23:12.373692 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.373656 4971 scope.go:122] "RemoveContainer" containerID="8cfc317abacc2cfa0a0e47cad54d1f730e60f07be5c38d1f3d6ece2a3ae93667" Nov 02 20:23:12.373964 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:23:12.373944 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cfc317abacc2cfa0a0e47cad54d1f730e60f07be5c38d1f3d6ece2a3ae93667\": container with ID starting with 8cfc317abacc2cfa0a0e47cad54d1f730e60f07be5c38d1f3d6ece2a3ae93667 not found: ID does not exist" containerID="8cfc317abacc2cfa0a0e47cad54d1f730e60f07be5c38d1f3d6ece2a3ae93667" Nov 02 20:23:12.374059 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.373970 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cfc317abacc2cfa0a0e47cad54d1f730e60f07be5c38d1f3d6ece2a3ae93667"} err="failed to get container status \"8cfc317abacc2cfa0a0e47cad54d1f730e60f07be5c38d1f3d6ece2a3ae93667\": rpc error: code = NotFound desc = could not find container \"8cfc317abacc2cfa0a0e47cad54d1f730e60f07be5c38d1f3d6ece2a3ae93667\": container with ID starting with 8cfc317abacc2cfa0a0e47cad54d1f730e60f07be5c38d1f3d6ece2a3ae93667 not found: ID does not exist" Nov 02 20:23:12.374059 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.373987 4971 scope.go:122] "RemoveContainer" containerID="1bc8538b7f0e5ca9cdb55985d07ca6f318a0bc5c31eaec131a3837c01795bea8" Nov 02 20:23:12.379800 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.379780 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913d5c7f-6064-4262-b624-b9f65afb7c66-kube-api-access-rxrzg\" (UniqueName: \"kubernetes.io/projected/913d5c7f-6064-4262-b624-b9f65afb7c66-kube-api-access-rxrzg\") pod \"913d5c7f-6064-4262-b624-b9f65afb7c66\" (UID: \"913d5c7f-6064-4262-b624-b9f65afb7c66\") " Nov 02 20:23:12.381571 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.381548 4971 scope.go:122] "RemoveContainer" containerID="1bc8538b7f0e5ca9cdb55985d07ca6f318a0bc5c31eaec131a3837c01795bea8" Nov 02 20:23:12.381845 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:23:12.381824 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc8538b7f0e5ca9cdb55985d07ca6f318a0bc5c31eaec131a3837c01795bea8\": container with ID starting with 1bc8538b7f0e5ca9cdb55985d07ca6f318a0bc5c31eaec131a3837c01795bea8 not found: ID does not exist" containerID="1bc8538b7f0e5ca9cdb55985d07ca6f318a0bc5c31eaec131a3837c01795bea8" Nov 02 20:23:12.381924 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.381844 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913d5c7f-6064-4262-b624-b9f65afb7c66-kube-api-access-rxrzg" pod "913d5c7f-6064-4262-b624-b9f65afb7c66" (UID: "913d5c7f-6064-4262-b624-b9f65afb7c66"). InnerVolumeSpecName "kube-api-access-rxrzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:12.381924 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.381854 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc8538b7f0e5ca9cdb55985d07ca6f318a0bc5c31eaec131a3837c01795bea8"} err="failed to get container status \"1bc8538b7f0e5ca9cdb55985d07ca6f318a0bc5c31eaec131a3837c01795bea8\": rpc error: code = NotFound desc = could not find container \"1bc8538b7f0e5ca9cdb55985d07ca6f318a0bc5c31eaec131a3837c01795bea8\": container with ID starting with 1bc8538b7f0e5ca9cdb55985d07ca6f318a0bc5c31eaec131a3837c01795bea8 not found: ID does not exist" Nov 02 20:23:12.381924 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.381875 4971 scope.go:122] "RemoveContainer" containerID="76aea97ad079382377056170bc1fc7ab282b5cdddb820ec81895da0811fed366" Nov 02 20:23:12.391665 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.391637 4971 scope.go:122] "RemoveContainer" containerID="76aea97ad079382377056170bc1fc7ab282b5cdddb820ec81895da0811fed366" Nov 02 20:23:12.391924 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:23:12.391906 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76aea97ad079382377056170bc1fc7ab282b5cdddb820ec81895da0811fed366\": container with ID starting with 76aea97ad079382377056170bc1fc7ab282b5cdddb820ec81895da0811fed366 not found: ID does not exist" containerID="76aea97ad079382377056170bc1fc7ab282b5cdddb820ec81895da0811fed366" Nov 02 20:23:12.391993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.391932 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76aea97ad079382377056170bc1fc7ab282b5cdddb820ec81895da0811fed366"} err="failed to get container status \"76aea97ad079382377056170bc1fc7ab282b5cdddb820ec81895da0811fed366\": rpc error: code = NotFound desc = could not find container \"76aea97ad079382377056170bc1fc7ab282b5cdddb820ec81895da0811fed366\": container with ID starting with 76aea97ad079382377056170bc1fc7ab282b5cdddb820ec81895da0811fed366 not found: ID does not exist" Nov 02 20:23:12.480823 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.480762 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxrzg\" (UniqueName: \"kubernetes.io/projected/913d5c7f-6064-4262-b624-b9f65afb7c66-kube-api-access-rxrzg\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:12.543331 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.543305 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-56rqk"] Nov 02 20:23:12.547119 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.547095 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-56rqk"] Nov 02 20:23:12.698558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.698532 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75f848775b-57bcc"] Nov 02 20:23:12.706495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.706471 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75f848775b-57bcc"] Nov 02 20:23:12.811954 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.811895 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fh9lt"] Nov 02 20:23:12.817542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:12.817519 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fh9lt"] Nov 02 20:23:13.864933 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:13.864893 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="913d5c7f-6064-4262-b624-b9f65afb7c66" path="/var/lib/kubelet/pods/913d5c7f-6064-4262-b624-b9f65afb7c66/volumes" Nov 02 20:23:13.865502 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:13.865388 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="973c7ab3-8cc7-4df1-8a7b-70a2288dd280" path="/var/lib/kubelet/pods/973c7ab3-8cc7-4df1-8a7b-70a2288dd280/volumes" Nov 02 20:23:13.865775 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:13.865752 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d75d87-17ea-452c-9f1e-b2554ed5b6c1" path="/var/lib/kubelet/pods/d6d75d87-17ea-452c-9f1e-b2554ed5b6c1/volumes" Nov 02 20:23:24.593781 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:24.593747 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:26.278782 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:26.278751 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-8ggqj" Nov 02 20:23:33.337031 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:23:33.336998 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:23:38.186467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:38.186433 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:23:49.249260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:23:49.249213 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:03.269723 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:24:03.269691 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:03.336157 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:24:03.336115 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:24:15.585226 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:24:15.585150 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:17.467682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:24:17.467649 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:24.934009 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:24:24.933929 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:31.907995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:24:31.907964 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:24:31.908441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:24:31.908215 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:24:33.337228 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:24:33.337193 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:24:47.379912 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:24:47.379855 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:24:47.626076 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:24:47.626039 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:03.338447 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:25:03.338397 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:25:05.266423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:05.266392 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:06.708569 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:06.708524 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:06.954631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:06.954600 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:08.689010 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.688980 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm"] Nov 02 20:25:08.689405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.689371 4971 cpu_manager.go:414] "containerMap: removing container" podUID="913d5c7f-6064-4262-b624-b9f65afb7c66" containerName="authorino" Nov 02 20:25:08.689405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.689383 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913d5c7f-6064-4262-b624-b9f65afb7c66" containerName="authorino" Nov 02 20:25:08.689405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.689394 4971 cpu_manager.go:414] "containerMap: removing container" podUID="d6d75d87-17ea-452c-9f1e-b2554ed5b6c1" containerName="authorino" Nov 02 20:25:08.689405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.689399 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d75d87-17ea-452c-9f1e-b2554ed5b6c1" containerName="authorino" Nov 02 20:25:08.689405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.689406 4971 cpu_manager.go:414] "containerMap: removing container" podUID="973c7ab3-8cc7-4df1-8a7b-70a2288dd280" containerName="authorino" Nov 02 20:25:08.689589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.689411 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="973c7ab3-8cc7-4df1-8a7b-70a2288dd280" containerName="authorino" Nov 02 20:25:08.689589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.689469 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d75d87-17ea-452c-9f1e-b2554ed5b6c1" containerName="authorino" Nov 02 20:25:08.689589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.689478 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="913d5c7f-6064-4262-b624-b9f65afb7c66" containerName="authorino" Nov 02 20:25:08.689589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.689486 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="973c7ab3-8cc7-4df1-8a7b-70a2288dd280" containerName="authorino" Nov 02 20:25:08.692051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.692032 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.741420 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.741398 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 20:25:08.788134 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.788107 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 20:25:08.788310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.788153 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rwxhc\"" Nov 02 20:25:08.788310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.788153 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 02 20:25:08.791335 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.791305 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm"] Nov 02 20:25:08.872937 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.872908 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b57b66-cb67-4bc2-ac17-c04b63c264d2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.873128 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.872944 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd8dp\" (UniqueName: \"kubernetes.io/projected/77b57b66-cb67-4bc2-ac17-c04b63c264d2-kube-api-access-sd8dp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.873128 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.873011 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b57b66-cb67-4bc2-ac17-c04b63c264d2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.873128 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.873054 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b57b66-cb67-4bc2-ac17-c04b63c264d2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.873282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.873199 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b57b66-cb67-4bc2-ac17-c04b63c264d2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.873282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.873257 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b57b66-cb67-4bc2-ac17-c04b63c264d2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.894158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.894129 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:08.973907 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.973853 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b57b66-cb67-4bc2-ac17-c04b63c264d2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.973907 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.973890 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b57b66-cb67-4bc2-ac17-c04b63c264d2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.974029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.973909 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd8dp\" (UniqueName: \"kubernetes.io/projected/77b57b66-cb67-4bc2-ac17-c04b63c264d2-kube-api-access-sd8dp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.974029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.974007 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b57b66-cb67-4bc2-ac17-c04b63c264d2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.974166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.974054 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b57b66-cb67-4bc2-ac17-c04b63c264d2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.974166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.974137 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b57b66-cb67-4bc2-ac17-c04b63c264d2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.974346 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.974323 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b57b66-cb67-4bc2-ac17-c04b63c264d2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.974475 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.974453 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b57b66-cb67-4bc2-ac17-c04b63c264d2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.974538 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.974483 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b57b66-cb67-4bc2-ac17-c04b63c264d2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.976336 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.976312 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b57b66-cb67-4bc2-ac17-c04b63c264d2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:08.976645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:08.976627 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b57b66-cb67-4bc2-ac17-c04b63c264d2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:09.031484 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:09.031456 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd8dp\" (UniqueName: \"kubernetes.io/projected/77b57b66-cb67-4bc2-ac17-c04b63c264d2-kube-api-access-sd8dp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm\" (UID: \"77b57b66-cb67-4bc2-ac17-c04b63c264d2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:09.303881 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:09.303804 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:09.666893 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:09.666867 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm"] Nov 02 20:25:09.668648 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:25:09.668623 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b57b66_cb67_4bc2_ac17_c04b63c264d2.slice/crio-11725099d2c18b99b785220b935b567f8aaa52bfb7432eedd058c6df667385ca WatchSource:0}: Error finding container 11725099d2c18b99b785220b935b567f8aaa52bfb7432eedd058c6df667385ca: Status 404 returned error can't find the container with id 11725099d2c18b99b785220b935b567f8aaa52bfb7432eedd058c6df667385ca Nov 02 20:25:09.669998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:09.669978 4971 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:25:09.920985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:09.920907 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" event={"ID":"77b57b66-cb67-4bc2-ac17-c04b63c264d2","Type":"ContainerStarted","Data":"11725099d2c18b99b785220b935b567f8aaa52bfb7432eedd058c6df667385ca"} Nov 02 20:25:11.931461 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:11.931422 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" event={"ID":"77b57b66-cb67-4bc2-ac17-c04b63c264d2","Type":"ContainerStarted","Data":"b75bb282c2e75f32add3ada79d810b595897a2be431cde661285c57cba9c1949"} Nov 02 20:25:12.042770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:12.042717 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" podStartSLOduration=2.553030794 podStartE2EDuration="4.042703084s" totalImagesPullingTime="1.48967229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:12.041917881 +0000 UTC m=+640.331574410" watchObservedRunningTime="2026-11-02 20:25:12.042703084 +0000 UTC m=+640.332359612" Nov 02 20:25:12.304584 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:12.304505 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:12.514433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:12.514397 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:15.429010 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.428977 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r"] Nov 02 20:25:15.432520 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.432502 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.482172 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.482145 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 02 20:25:15.488879 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.488856 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r"] Nov 02 20:25:15.624465 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.624435 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:15.626599 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.626567 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.626599 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.626598 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.626758 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.626632 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbzcw\" (UniqueName: \"kubernetes.io/projected/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-kube-api-access-rbzcw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.626796 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.626767 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.626830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.626805 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.626866 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.626826 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.727811 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.727753 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.727811 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.727791 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.727811 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.727810 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.728008 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.727831 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.728008 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.727848 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.728008 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.727878 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbzcw\" (UniqueName: \"kubernetes.io/projected/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-kube-api-access-rbzcw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.728284 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.728243 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.728398 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.728280 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.728460 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.728405 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.730438 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.730415 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.730546 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.730530 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:15.787200 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:15.787171 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbzcw\" (UniqueName: \"kubernetes.io/projected/1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a-kube-api-access-rbzcw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4wj6r\" (UID: \"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:16.044644 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:16.044569 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:16.410884 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:16.410854 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r"] Nov 02 20:25:16.412594 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:25:16.412567 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2ab6f3_51c8_47b6_bbd8_d0fde651c16a.slice/crio-450076c7469245a829f352615a44994cbd4be7aa75e13c0769f197b050cbf7bf WatchSource:0}: Error finding container 450076c7469245a829f352615a44994cbd4be7aa75e13c0769f197b050cbf7bf: Status 404 returned error can't find the container with id 450076c7469245a829f352615a44994cbd4be7aa75e13c0769f197b050cbf7bf Nov 02 20:25:16.958268 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:16.958226 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" event={"ID":"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a","Type":"ContainerStarted","Data":"450076c7469245a829f352615a44994cbd4be7aa75e13c0769f197b050cbf7bf"} Nov 02 20:25:17.964109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:17.964065 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" event={"ID":"1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a","Type":"ContainerStarted","Data":"2d350a13ffaa237594babff029052cd6ae8b245396b13dacbff30eaf57be310f"} Nov 02 20:25:18.045264 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.045235 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:18.076452 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.076407 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" podStartSLOduration=2.424780035 podStartE2EDuration="3.076395172s" totalImagesPullingTime="651.615137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:18.074924971 +0000 UTC m=+646.364581500" watchObservedRunningTime="2026-11-02 20:25:18.076395172 +0000 UTC m=+646.366051700" Nov 02 20:25:18.373467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.373442 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn"] Nov 02 20:25:18.376704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.376686 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.426005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.425972 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 02 20:25:18.450736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.450702 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.450875 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.450738 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.450875 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.450855 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.450995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.450903 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.450995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.450974 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cfww\" (UniqueName: \"kubernetes.io/projected/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-kube-api-access-5cfww\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.451130 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.451000 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.474356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.474331 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn"] Nov 02 20:25:18.551625 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.551598 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.551764 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.551639 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.551764 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.551665 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.551764 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.551686 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.551764 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.551718 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cfww\" (UniqueName: \"kubernetes.io/projected/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-kube-api-access-5cfww\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.551764 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.551745 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.552026 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.551979 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.552115 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.552035 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.552166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.552124 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.554058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.554030 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.554416 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.554396 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.574054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.574034 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:18.609466 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.609443 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cfww\" (UniqueName: \"kubernetes.io/projected/f0316745-a138-4b26-9f2a-a9f5ffd9d97c-kube-api-access-5cfww\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hc8gn\" (UID: \"f0316745-a138-4b26-9f2a-a9f5ffd9d97c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:18.689722 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:18.689668 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:19.057038 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:19.057013 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn"] Nov 02 20:25:19.058974 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:25:19.058944 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0316745_a138_4b26_9f2a_a9f5ffd9d97c.slice/crio-896d5a5d00022e00422fb9878f006600a2acbaaa19c5532c9d0ca27cbd9a137d WatchSource:0}: Error finding container 896d5a5d00022e00422fb9878f006600a2acbaaa19c5532c9d0ca27cbd9a137d: Status 404 returned error can't find the container with id 896d5a5d00022e00422fb9878f006600a2acbaaa19c5532c9d0ca27cbd9a137d Nov 02 20:25:19.304687 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:19.304661 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:19.323975 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:19.323907 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:19.975470 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:19.975438 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" event={"ID":"f0316745-a138-4b26-9f2a-a9f5ffd9d97c","Type":"ContainerStarted","Data":"d6161ba83f6f62960794b0ef58a98d806d669f89553477648c4941873d4e9f51"} Nov 02 20:25:19.975643 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:19.975493 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" event={"ID":"f0316745-a138-4b26-9f2a-a9f5ffd9d97c","Type":"ContainerStarted","Data":"896d5a5d00022e00422fb9878f006600a2acbaaa19c5532c9d0ca27cbd9a137d"} Nov 02 20:25:19.986702 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:19.986683 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm" Nov 02 20:25:20.088316 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:20.088239 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" podStartSLOduration=1.3146963999999999 podStartE2EDuration="2.088227024s" totalImagesPullingTime="773.530624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:20.086218423 +0000 UTC m=+648.375874954" watchObservedRunningTime="2026-11-02 20:25:20.088227024 +0000 UTC m=+648.377883552" Nov 02 20:25:20.690618 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:20.690578 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:23.243782 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:23.243750 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:25.258055 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:25.258025 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:25.505316 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:25.505289 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:26.044909 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:26.044877 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:26.058456 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:26.058429 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:26.261313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:26.261281 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:27.027116 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:27.027069 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4wj6r" Nov 02 20:25:28.689854 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:28.689822 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:28.702093 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:28.702054 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:29.036603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:29.036531 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" Nov 02 20:25:32.985789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:32.985755 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt"] Nov 02 20:25:32.992094 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:32.992059 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.042515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.042489 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 02 20:25:33.070491 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.070459 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.070611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.070498 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.070611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.070519 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6dxq\" (UniqueName: \"kubernetes.io/projected/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-kube-api-access-h6dxq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.070611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.070594 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.070713 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.070668 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.070750 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.070715 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.091330 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.091306 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt"] Nov 02 20:25:33.171900 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.171862 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.172040 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.171925 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.172040 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.171972 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6dxq\" (UniqueName: \"kubernetes.io/projected/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-kube-api-access-h6dxq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.172040 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.172000 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.172040 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.172037 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.172441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.172071 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.172441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.172394 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.172639 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.172619 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.173001 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.172889 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.174774 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.174744 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.175006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.174984 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.186785 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.186763 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:33.232744 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.232719 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6dxq\" (UniqueName: \"kubernetes.io/projected/5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361-kube-api-access-h6dxq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt\" (UID: \"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.318442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.318382 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:33.346693 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:25:33.346653 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:25:33.687646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:33.687619 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt"] Nov 02 20:25:33.689047 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:25:33.689024 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f4bc85d_53a1_44c4_a8bd_88b7b5b6e361.slice/crio-bb440dddc44ad8e3ad5d181cf879b70903aea1297e6a05dfe60d6b0371d168a2 WatchSource:0}: Error finding container bb440dddc44ad8e3ad5d181cf879b70903aea1297e6a05dfe60d6b0371d168a2: Status 404 returned error can't find the container with id bb440dddc44ad8e3ad5d181cf879b70903aea1297e6a05dfe60d6b0371d168a2 Nov 02 20:25:34.052405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:34.052323 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" event={"ID":"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361","Type":"ContainerStarted","Data":"bb440dddc44ad8e3ad5d181cf879b70903aea1297e6a05dfe60d6b0371d168a2"} Nov 02 20:25:35.058414 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:35.058382 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" event={"ID":"5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361","Type":"ContainerStarted","Data":"e6ff3a847aae32a781df0938c9ce8ef75b94f951112c7e9851ae812f53ff1074"} Nov 02 20:25:35.170801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:35.170757 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podStartSLOduration=2.297958903 podStartE2EDuration="3.17074278s" totalImagesPullingTime="872.783877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:35.169293223 +0000 UTC m=+663.458949751" watchObservedRunningTime="2026-11-02 20:25:35.17074278 +0000 UTC m=+663.460399308" Nov 02 20:25:35.318790 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:35.318718 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:38.351803 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.351763 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk"] Nov 02 20:25:38.356895 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.356871 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.406342 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.406320 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dg9qz\"" Nov 02 20:25:38.415656 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.415626 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.415740 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.415662 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.415740 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.415680 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc5m7\" (UniqueName: \"kubernetes.io/projected/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-kube-api-access-tc5m7\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.415819 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.415760 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.415819 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.415796 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.415886 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.415822 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.415919 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.415897 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.415952 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.415921 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.415988 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.415976 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.456346 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.456322 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk"] Nov 02 20:25:38.516765 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.516737 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.516935 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.516773 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.516935 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.516793 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc5m7\" (UniqueName: \"kubernetes.io/projected/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-kube-api-access-tc5m7\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.516935 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.516811 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.516935 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.516829 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.516935 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.516863 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.516935 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.516903 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.517291 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.516928 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.517291 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.517061 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.517291 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.517216 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.517291 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.517271 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.517550 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.517489 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.517672 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.517647 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.517774 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.517748 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.520245 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.520187 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.520356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.520259 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.574958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.574931 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.575115 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.575057 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc5m7\" (UniqueName: \"kubernetes.io/projected/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-kube-api-access-tc5m7\") pod \"maas-default-gateway-openshift-default-85544d4cd7-k98hk\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:38.670682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:38.670625 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:39.086460 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:39.086434 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk"] Nov 02 20:25:39.087767 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:25:39.087739 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4071971f_4b3b_45f8_9ba4_ea8eb9f6e1b9.slice/crio-a6cdfc5bfc66356913ba28baeccc0cd99191c96ee5c5a9facc6a27cbf25423ff WatchSource:0}: Error finding container a6cdfc5bfc66356913ba28baeccc0cd99191c96ee5c5a9facc6a27cbf25423ff: Status 404 returned error can't find the container with id a6cdfc5bfc66356913ba28baeccc0cd99191c96ee5c5a9facc6a27cbf25423ff Nov 02 20:25:39.089927 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:39.089897 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:25:39.090002 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:39.089973 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:25:39.090034 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:39.090010 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:25:40.083846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:40.083808 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" event={"ID":"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9","Type":"ContainerStarted","Data":"830e90b89c9d409b8dcdb929c52925940cbce950f2734dccedf5a4b02824c8cf"} Nov 02 20:25:40.084353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:40.083865 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" event={"ID":"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9","Type":"ContainerStarted","Data":"a6cdfc5bfc66356913ba28baeccc0cd99191c96ee5c5a9facc6a27cbf25423ff"} Nov 02 20:25:40.303303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:40.303243 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" podStartSLOduration=2.30322905 podStartE2EDuration="2.30322905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:40.255730782 +0000 UTC m=+668.545387311" watchObservedRunningTime="2026-11-02 20:25:40.30322905 +0000 UTC m=+668.592885577" Nov 02 20:25:40.671553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:40.671514 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:40.673068 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:40.673044 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-k98hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 20:25:40.673212 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:40.673103 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" podUID="4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 20:25:41.671445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:41.671353 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-k98hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 20:25:41.671908 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:41.671438 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" podUID="4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 20:25:42.671562 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:42.671519 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-k98hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 20:25:42.672009 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:42.671577 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" podUID="4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 20:25:43.319016 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:43.318985 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:43.357566 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:43.357537 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:43.671337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:43.671305 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-k98hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 20:25:43.671518 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:43.671354 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" podUID="4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 20:25:44.114258 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:44.114227 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:25:44.671565 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:44.671527 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-k98hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 20:25:44.671749 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:44.671595 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" podUID="4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 20:25:45.671214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:45.671180 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-k98hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 20:25:45.671589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:45.671242 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" podUID="4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 20:25:47.316408 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:47.316381 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:47.316839 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:47.316795 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:47.317508 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:47.317491 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:25:53.362778 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.362748 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr"] Nov 02 20:25:53.366661 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.366645 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.464298 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.464264 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.464448 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.464333 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.464520 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.464440 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.464520 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.464492 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.464632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.464554 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.464632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.464590 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdcmq\" (UniqueName: \"kubernetes.io/projected/e271f874-deb0-487e-a4bc-ee0f520bb501-kube-api-access-hdcmq\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.464789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.464632 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.464789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.464683 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.464789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.464717 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e271f874-deb0-487e-a4bc-ee0f520bb501-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.465513 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.465478 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr"] Nov 02 20:25:53.565538 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.565507 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.565669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.565546 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.565669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.565598 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.565669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.565628 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdcmq\" (UniqueName: \"kubernetes.io/projected/e271f874-deb0-487e-a4bc-ee0f520bb501-kube-api-access-hdcmq\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.565788 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.565665 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.565788 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.565691 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.565897 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.565845 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e271f874-deb0-487e-a4bc-ee0f520bb501-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.565952 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.565908 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.566005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.565982 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.566063 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.566035 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.566063 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.566051 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.566239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.566042 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.566326 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.566307 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.566617 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.566591 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e271f874-deb0-487e-a4bc-ee0f520bb501-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.568361 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.568340 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.568455 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.568376 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.623560 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.623490 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.623704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.623684 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdcmq\" (UniqueName: \"kubernetes.io/projected/e271f874-deb0-487e-a4bc-ee0f520bb501-kube-api-access-hdcmq\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c4bhr\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:53.680460 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:53.680440 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:54.097774 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:54.097748 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr"] Nov 02 20:25:54.099512 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:25:54.099478 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode271f874_deb0_487e_a4bc_ee0f520bb501.slice/crio-679444b3369cc92f89a87c833222dfd0c69bfe257f477e052a17c24b38f9d4bd WatchSource:0}: Error finding container 679444b3369cc92f89a87c833222dfd0c69bfe257f477e052a17c24b38f9d4bd: Status 404 returned error can't find the container with id 679444b3369cc92f89a87c833222dfd0c69bfe257f477e052a17c24b38f9d4bd Nov 02 20:25:54.101662 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:54.101627 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:25:54.101741 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:54.101692 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:25:54.101741 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:54.101723 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:25:54.157779 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:54.157753 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" event={"ID":"e271f874-deb0-487e-a4bc-ee0f520bb501","Type":"ContainerStarted","Data":"679444b3369cc92f89a87c833222dfd0c69bfe257f477e052a17c24b38f9d4bd"} Nov 02 20:25:55.164214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:55.164179 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" event={"ID":"e271f874-deb0-487e-a4bc-ee0f520bb501","Type":"ContainerStarted","Data":"03b86e5c7290400c2b5186ef67d2287dda37bc042fe5423787f9086982991b46"} Nov 02 20:25:55.680970 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:55.680935 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:25:55.682631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:55.682605 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c4bhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 20:25:55.682790 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:55.682666 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" podUID="e271f874-deb0-487e-a4bc-ee0f520bb501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 20:25:56.681550 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:56.681513 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c4bhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 20:25:56.681947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:56.681561 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" podUID="e271f874-deb0-487e-a4bc-ee0f520bb501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 20:25:57.681291 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:57.681260 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c4bhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 20:25:57.681459 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:57.681310 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" podUID="e271f874-deb0-487e-a4bc-ee0f520bb501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 20:25:57.849138 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:57.849060 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" podStartSLOduration=4.849045903 podStartE2EDuration="4.849045903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:55.347171543 +0000 UTC m=+683.636828084" watchObservedRunningTime="2026-11-02 20:25:57.849045903 +0000 UTC m=+686.138702434" Nov 02 20:25:57.850834 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:57.850815 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85d44d9c5c-jglfz"] Nov 02 20:25:57.854575 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:57.854553 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85d44d9c5c-jglfz" Nov 02 20:25:57.906956 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:57.906928 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85d44d9c5c-jglfz"] Nov 02 20:25:57.951571 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:57.951484 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 20:25:57.951713 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:57.951582 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gvxgg\"" Nov 02 20:25:58.008263 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:58.008235 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/585cb390-ad62-4295-8f12-04179043e96c-tls-cert\") pod \"authorino-85d44d9c5c-jglfz\" (UID: \"585cb390-ad62-4295-8f12-04179043e96c\") " pod="kuadrant-system/authorino-85d44d9c5c-jglfz" Nov 02 20:25:58.008409 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:58.008346 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4hnv\" (UniqueName: \"kubernetes.io/projected/585cb390-ad62-4295-8f12-04179043e96c-kube-api-access-x4hnv\") pod \"authorino-85d44d9c5c-jglfz\" (UID: \"585cb390-ad62-4295-8f12-04179043e96c\") " pod="kuadrant-system/authorino-85d44d9c5c-jglfz" Nov 02 20:25:58.109758 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:58.109724 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/585cb390-ad62-4295-8f12-04179043e96c-tls-cert\") pod \"authorino-85d44d9c5c-jglfz\" (UID: \"585cb390-ad62-4295-8f12-04179043e96c\") " pod="kuadrant-system/authorino-85d44d9c5c-jglfz" Nov 02 20:25:58.109924 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:58.109897 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hnv\" (UniqueName: \"kubernetes.io/projected/585cb390-ad62-4295-8f12-04179043e96c-kube-api-access-x4hnv\") pod \"authorino-85d44d9c5c-jglfz\" (UID: \"585cb390-ad62-4295-8f12-04179043e96c\") " pod="kuadrant-system/authorino-85d44d9c5c-jglfz" Nov 02 20:25:58.112489 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:58.112462 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/585cb390-ad62-4295-8f12-04179043e96c-tls-cert\") pod \"authorino-85d44d9c5c-jglfz\" (UID: \"585cb390-ad62-4295-8f12-04179043e96c\") " pod="kuadrant-system/authorino-85d44d9c5c-jglfz" Nov 02 20:25:58.154411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:58.154382 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:58.167420 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:58.167395 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4hnv\" (UniqueName: \"kubernetes.io/projected/585cb390-ad62-4295-8f12-04179043e96c-kube-api-access-x4hnv\") pod \"authorino-85d44d9c5c-jglfz\" (UID: \"585cb390-ad62-4295-8f12-04179043e96c\") " pod="kuadrant-system/authorino-85d44d9c5c-jglfz" Nov 02 20:25:58.168209 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:58.168185 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85d44d9c5c-jglfz" Nov 02 20:25:58.407230 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:58.407198 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:25:58.557952 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:58.557923 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85d44d9c5c-jglfz"] Nov 02 20:25:58.560253 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:25:58.560221 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod585cb390_ad62_4295_8f12_04179043e96c.slice/crio-1d540117631982c12b6d8813798610c41fbb078d94c1e4c54c4ac6806bdc8187 WatchSource:0}: Error finding container 1d540117631982c12b6d8813798610c41fbb078d94c1e4c54c4ac6806bdc8187: Status 404 returned error can't find the container with id 1d540117631982c12b6d8813798610c41fbb078d94c1e4c54c4ac6806bdc8187 Nov 02 20:25:58.681072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:58.681036 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c4bhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 20:25:58.681266 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:58.681116 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" podUID="e271f874-deb0-487e-a4bc-ee0f520bb501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 20:25:59.184463 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:59.184430 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85d44d9c5c-jglfz" event={"ID":"585cb390-ad62-4295-8f12-04179043e96c","Type":"ContainerStarted","Data":"1d540117631982c12b6d8813798610c41fbb078d94c1e4c54c4ac6806bdc8187"} Nov 02 20:25:59.680856 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:59.680821 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c4bhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 20:25:59.681051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:25:59.680878 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" podUID="e271f874-deb0-487e-a4bc-ee0f520bb501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 20:26:00.681685 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:26:00.681651 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c4bhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 20:26:00.682170 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:26:00.681709 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" podUID="e271f874-deb0-487e-a4bc-ee0f520bb501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 20:26:02.293019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:26:02.292988 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:26:02.293506 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:26:02.293445 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:26:02.294261 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:26:02.294237 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:26:03.208053 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:26:03.208019 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85d44d9c5c-jglfz" event={"ID":"585cb390-ad62-4295-8f12-04179043e96c","Type":"ContainerStarted","Data":"84773fad286b890d9cfa117884810d6040940fb2de3d2ac4ff31ba86340c733d"} Nov 02 20:26:03.318841 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:26:03.318789 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-85d44d9c5c-jglfz" podStartSLOduration=2.415607078 podStartE2EDuration="6.31877518s" totalImagesPullingTime="3.903168102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:03.317218061 +0000 UTC m=+691.606874591" watchObservedRunningTime="2026-11-02 20:26:03.31877518 +0000 UTC m=+691.608431754" Nov 02 20:26:03.353954 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:26:03.353883 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:26:07.563860 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:26:07.563814 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:07.813438 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:26:07.813400 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:25.615395 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:26:25.615356 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:30.912006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:26:30.911971 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:31.395744 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:26:31.395712 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:33.037407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:26:33.037373 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:33.301004 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:26:33.300919 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:33.338748 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:26:33.338714 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:26:34.670849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:26:34.670813 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:26:34.945609 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:26:34.945536 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:01.359640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:01.359603 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:01.616739 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:01.616664 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:01.950687 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:01.950603 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:02.053540 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.053508 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9"] Nov 02 20:27:02.057407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.057383 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.128353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.128324 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9"] Nov 02 20:27:02.153721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.153693 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a5cad4-openshift-default-dockercfg-phnwd\"" Nov 02 20:27:02.191675 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.191650 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.191788 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.191694 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.191833 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.191798 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.191877 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.191841 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.191877 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.191859 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.191959 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.191949 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.192001 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.191979 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.192045 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.192017 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2nsp\" (UniqueName: \"kubernetes.io/projected/41baf9c8-0dc6-4e97-958c-70391cbeb98b-kube-api-access-x2nsp\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.192045 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.192039 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.197727 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.197705 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:02.293265 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.293197 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.293265 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.293256 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2nsp\" (UniqueName: \"kubernetes.io/projected/41baf9c8-0dc6-4e97-958c-70391cbeb98b-kube-api-access-x2nsp\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.293441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.293291 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.293441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.293330 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.293441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.293384 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.293597 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.293442 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.293597 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.293488 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.293597 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.293516 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.293797 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.293708 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.294040 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.293993 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.294484 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.294458 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.294587 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.294556 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.294587 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.294575 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.294825 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.294803 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.296257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.296238 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.296859 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.296841 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.356282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.356261 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.356626 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.356608 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2nsp\" (UniqueName: \"kubernetes.io/projected/41baf9c8-0dc6-4e97-958c-70391cbeb98b-kube-api-access-x2nsp\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.370064 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.370002 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:02.566703 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.566609 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:02.792070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.792039 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9"] Nov 02 20:27:02.793668 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:27:02.793625 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41baf9c8_0dc6_4e97_958c_70391cbeb98b.slice/crio-803ef52e54a2fc3bb1abbbb9fa20d9898bf9068b488dd5d1da88301ba133fc76 WatchSource:0}: Error finding container 803ef52e54a2fc3bb1abbbb9fa20d9898bf9068b488dd5d1da88301ba133fc76: Status 404 returned error can't find the container with id 803ef52e54a2fc3bb1abbbb9fa20d9898bf9068b488dd5d1da88301ba133fc76 Nov 02 20:27:02.796145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.796112 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:02.796221 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.796180 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:02.796221 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.796206 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:02.820421 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.820349 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:02.972800 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.972768 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7"] Nov 02 20:27:02.976624 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:02.976598 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.027974 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.027949 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-694cee-openshift-default-dockercfg-tv8jc\"" Nov 02 20:27:03.036359 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.036227 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7"] Nov 02 20:27:03.102553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.102526 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.102700 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.102570 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.102700 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.102683 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.102808 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.102717 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94xb4\" (UniqueName: \"kubernetes.io/projected/0e0eeb03-8f43-4579-bdb9-66e80f676abb-kube-api-access-94xb4\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.102808 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.102747 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.102913 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.102852 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.102970 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.102922 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.103025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.102979 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.103108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.103020 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.124908 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.124859 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:03.203834 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.203800 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.203997 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.203859 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.203997 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.203888 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.203997 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.203935 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.204145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.204071 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94xb4\" (UniqueName: \"kubernetes.io/projected/0e0eeb03-8f43-4579-bdb9-66e80f676abb-kube-api-access-94xb4\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.204207 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.204147 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.204261 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.204206 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.204313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.204257 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.204367 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.204322 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.204367 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.204344 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.204467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.204449 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.204594 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.204531 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.204771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.204675 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.205228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.205164 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.206447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.206422 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.207129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.207106 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.263537 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.263510 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.263625 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.263572 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94xb4\" (UniqueName: \"kubernetes.io/projected/0e0eeb03-8f43-4579-bdb9-66e80f676abb-kube-api-access-94xb4\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.279735 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.279711 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22"] Nov 02 20:27:03.284711 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.284695 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.289055 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.289036 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:03.340111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.335032 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-c49693-openshift-default-dockercfg-cq675\"" Nov 02 20:27:03.350125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.346571 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22"] Nov 02 20:27:03.361733 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:27:03.361650 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:27:03.374807 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.374775 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:03.406923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.406893 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-credential-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.407048 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.406934 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62qs7\" (UniqueName: \"kubernetes.io/projected/413b0cf5-5c2e-4983-8437-7f9d2e899405-kube-api-access-62qs7\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.407048 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.406954 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-data\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.407197 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.407130 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-podinfo\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.407255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.407239 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413b0cf5-5c2e-4983-8437-7f9d2e899405-istiod-ca-cert\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.407340 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.407326 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-workload-certs\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.407413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.407392 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-workload-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.407459 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.407445 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-envoy\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.407521 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.407463 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-token\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.508606 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.508580 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-envoy\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.508606 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.508608 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-token\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.508786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.508626 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-credential-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.508786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.508649 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62qs7\" (UniqueName: \"kubernetes.io/projected/413b0cf5-5c2e-4983-8437-7f9d2e899405-kube-api-access-62qs7\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.508786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.508668 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-data\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.508786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.508714 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-podinfo\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.508786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.508775 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413b0cf5-5c2e-4983-8437-7f9d2e899405-istiod-ca-cert\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.509028 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.508829 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-workload-certs\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.509028 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.508877 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-workload-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.509184 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.509046 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-credential-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.509184 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.509143 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-data\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.509316 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.509292 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-workload-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.509379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.509320 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-workload-certs\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.509581 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.509554 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413b0cf5-5c2e-4983-8437-7f9d2e899405-istiod-ca-cert\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.511192 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.511176 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-envoy\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.511252 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.511225 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-podinfo\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.518979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.518949 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" event={"ID":"41baf9c8-0dc6-4e97-958c-70391cbeb98b","Type":"ContainerStarted","Data":"ce529c7e4d15bf92455208c281ff7dcf850d819a17919501a8153c17fc9ab454"} Nov 02 20:27:03.519077 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.519011 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" event={"ID":"41baf9c8-0dc6-4e97-958c-70391cbeb98b","Type":"ContainerStarted","Data":"803ef52e54a2fc3bb1abbbb9fa20d9898bf9068b488dd5d1da88301ba133fc76"} Nov 02 20:27:03.566532 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.566511 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-token\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.566863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.566846 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62qs7\" (UniqueName: \"kubernetes.io/projected/413b0cf5-5c2e-4983-8437-7f9d2e899405-kube-api-access-62qs7\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.602327 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.602291 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:03.625221 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.625188 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:03.635576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.635523 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" podStartSLOduration=1.635503384 podStartE2EDuration="1.635503384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:03.63235233 +0000 UTC m=+751.922008858" watchObservedRunningTime="2026-11-02 20:27:03.635503384 +0000 UTC m=+751.925159916" Nov 02 20:27:03.680339 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.680295 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7"] Nov 02 20:27:03.687665 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.687629 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:03.687776 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.687705 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:03.687776 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.687747 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:03.877430 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.877356 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:03.976739 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.976714 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22"] Nov 02 20:27:03.977795 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:27:03.977764 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413b0cf5_5c2e_4983_8437_7f9d2e899405.slice/crio-b5a5dbbd3f8974d95215d74f344be960d4bbbbc4ee86cbf71113c965e8b4ab03 WatchSource:0}: Error finding container b5a5dbbd3f8974d95215d74f344be960d4bbbbc4ee86cbf71113c965e8b4ab03: Status 404 returned error can't find the container with id b5a5dbbd3f8974d95215d74f344be960d4bbbbc4ee86cbf71113c965e8b4ab03 Nov 02 20:27:03.980457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.980403 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:03.980558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.980503 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:03.980558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:03.980552 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:04.370419 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:04.370385 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:04.375751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:04.375727 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:04.526393 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:04.526358 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" event={"ID":"413b0cf5-5c2e-4983-8437-7f9d2e899405","Type":"ContainerStarted","Data":"327b455ca520a38bbc20f7043c200b3fc50952ad9e43941dacd42e699668857e"} Nov 02 20:27:04.526561 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:04.526418 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" event={"ID":"413b0cf5-5c2e-4983-8437-7f9d2e899405","Type":"ContainerStarted","Data":"b5a5dbbd3f8974d95215d74f344be960d4bbbbc4ee86cbf71113c965e8b4ab03"} Nov 02 20:27:04.528003 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:04.527973 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" event={"ID":"0e0eeb03-8f43-4579-bdb9-66e80f676abb","Type":"ContainerStarted","Data":"c6bce39071dea22b90e40e0a91e3a1cd7bf1f4142c20aad00c5a8b11a40924b8"} Nov 02 20:27:04.528177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:04.528020 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" event={"ID":"0e0eeb03-8f43-4579-bdb9-66e80f676abb","Type":"ContainerStarted","Data":"ac1a002f50ab871ec7e7a91d185c429ad58763136e4e701bbde83922c907b2c1"} Nov 02 20:27:04.528177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:04.528113 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:04.529137 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:04.529120 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:04.602760 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:04.602729 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:04.640771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:04.640679 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" podStartSLOduration=1.640666139 podStartE2EDuration="1.640666139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:04.639440255 +0000 UTC m=+752.929096783" watchObservedRunningTime="2026-11-02 20:27:04.640666139 +0000 UTC m=+752.930322657" Nov 02 20:27:04.764984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:04.764920 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" podStartSLOduration=2.764899867 podStartE2EDuration="2.764899867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:04.762761686 +0000 UTC m=+753.052418216" watchObservedRunningTime="2026-11-02 20:27:04.764899867 +0000 UTC m=+753.054556396" Nov 02 20:27:05.289622 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:05.289592 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:05.294615 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:05.294595 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:05.533748 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:05.533719 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:05.534542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:05.534524 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:05.607756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:05.607738 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:06.538347 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:06.538319 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:06.539289 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:06.539272 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:06.728705 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:06.728673 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:06.980662 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:06.980627 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:07.515874 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:07.515840 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:07.806950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:07.806875 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:08.224537 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:08.224504 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:08.477843 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:08.477772 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:08.902895 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:08.902845 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:09.157166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:09.157015 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:09.554156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:09.554064 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:09.801993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:09.801964 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:11.228590 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:11.228563 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:11.537591 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:11.537488 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:11.787291 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:11.787260 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:12.045442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:12.045413 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:12.148940 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:12.148909 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs"] Nov 02 20:27:12.153027 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:12.153001 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" Nov 02 20:27:12.254843 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:12.254815 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-30ee89-dockercfg-qqj5d\"" Nov 02 20:27:12.261125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:12.261097 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-30ee89\"" Nov 02 20:27:12.266400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:12.266374 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs"] Nov 02 20:27:12.297522 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:12.297419 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs\" (UID: \"5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" Nov 02 20:27:12.297690 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:12.297588 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p5ng\" (UniqueName: \"kubernetes.io/projected/5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4-kube-api-access-5p5ng\") pod \"payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs\" (UID: \"5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" Nov 02 20:27:12.398169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:12.398136 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p5ng\" (UniqueName: \"kubernetes.io/projected/5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4-kube-api-access-5p5ng\") pod \"payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs\" (UID: \"5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" Nov 02 20:27:12.398320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:12.398219 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs\" (UID: \"5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" Nov 02 20:27:12.398840 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:12.398822 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs\" (UID: \"5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" Nov 02 20:27:12.456167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:12.456135 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p5ng\" (UniqueName: \"kubernetes.io/projected/5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4-kube-api-access-5p5ng\") pod \"payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs\" (UID: \"5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" Nov 02 20:27:12.466985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:12.466964 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" Nov 02 20:27:12.830232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:12.830204 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs"] Nov 02 20:27:12.831743 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:27:12.831710 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccd73f1_7600_4593_8ee4_f6c00d6ea0f4.slice/crio-86bee941de3dfe309658c4104bfe0a74cb8f934c2477f9c8b8a90c214af10046 WatchSource:0}: Error finding container 86bee941de3dfe309658c4104bfe0a74cb8f934c2477f9c8b8a90c214af10046: Status 404 returned error can't find the container with id 86bee941de3dfe309658c4104bfe0a74cb8f934c2477f9c8b8a90c214af10046 Nov 02 20:27:13.320443 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:13.320413 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:13.568122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:13.568060 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:13.574593 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:13.574526 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" event={"ID":"5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4","Type":"ContainerStarted","Data":"86bee941de3dfe309658c4104bfe0a74cb8f934c2477f9c8b8a90c214af10046"} Nov 02 20:27:16.593836 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:16.593801 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" event={"ID":"5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4","Type":"ContainerStarted","Data":"3ff6e63ca94b047be44ad0a31a8f5c4c01d23d64769a882a6c104efdb78c184e"} Nov 02 20:27:16.594206 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:16.593969 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" Nov 02 20:27:16.703116 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:16.703001 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" podStartSLOduration=1.128404224 podStartE2EDuration="4.70298882s" totalImagesPullingTime="3.574584596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:16.702229501 +0000 UTC m=+764.991886028" watchObservedRunningTime="2026-11-02 20:27:16.70298882 +0000 UTC m=+764.992645347" Nov 02 20:27:16.772597 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:16.772560 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:17.014878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.014804 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:17.081135 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.081098 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn"] Nov 02 20:27:17.084865 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.084847 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" Nov 02 20:27:17.137241 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.137217 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-a5cad4\"" Nov 02 20:27:17.187108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.185832 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-a5cad4-dockercfg-2xk8x\"" Nov 02 20:27:17.194055 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.194022 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn"] Nov 02 20:27:17.229823 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.229794 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj"] Nov 02 20:27:17.233554 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.233536 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" Nov 02 20:27:17.245803 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.245781 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8daff05-2927-4e7b-95a4-201024607e67-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn\" (UID: \"e8daff05-2927-4e7b-95a4-201024607e67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" Nov 02 20:27:17.245908 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.245837 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t99kx\" (UniqueName: \"kubernetes.io/projected/e8daff05-2927-4e7b-95a4-201024607e67-kube-api-access-t99kx\") pod \"payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn\" (UID: \"e8daff05-2927-4e7b-95a4-201024607e67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" Nov 02 20:27:17.288142 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.288056 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj"] Nov 02 20:27:17.346972 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.346600 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee36cd67-b157-4fda-bd62-34ba6cd9460e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj\" (UID: \"ee36cd67-b157-4fda-bd62-34ba6cd9460e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" Nov 02 20:27:17.346972 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.346674 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8daff05-2927-4e7b-95a4-201024607e67-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn\" (UID: \"e8daff05-2927-4e7b-95a4-201024607e67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" Nov 02 20:27:17.346972 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.346730 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5pqf\" (UniqueName: \"kubernetes.io/projected/ee36cd67-b157-4fda-bd62-34ba6cd9460e-kube-api-access-t5pqf\") pod \"payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj\" (UID: \"ee36cd67-b157-4fda-bd62-34ba6cd9460e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" Nov 02 20:27:17.346972 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.346804 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99kx\" (UniqueName: \"kubernetes.io/projected/e8daff05-2927-4e7b-95a4-201024607e67-kube-api-access-t99kx\") pod \"payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn\" (UID: \"e8daff05-2927-4e7b-95a4-201024607e67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" Nov 02 20:27:17.348061 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.348036 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8daff05-2927-4e7b-95a4-201024607e67-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn\" (UID: \"e8daff05-2927-4e7b-95a4-201024607e67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" Nov 02 20:27:17.404814 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.404787 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t99kx\" (UniqueName: \"kubernetes.io/projected/e8daff05-2927-4e7b-95a4-201024607e67-kube-api-access-t99kx\") pod \"payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn\" (UID: \"e8daff05-2927-4e7b-95a4-201024607e67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" Nov 02 20:27:17.447623 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.447602 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee36cd67-b157-4fda-bd62-34ba6cd9460e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj\" (UID: \"ee36cd67-b157-4fda-bd62-34ba6cd9460e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" Nov 02 20:27:17.447726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.447634 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5pqf\" (UniqueName: \"kubernetes.io/projected/ee36cd67-b157-4fda-bd62-34ba6cd9460e-kube-api-access-t5pqf\") pod \"payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj\" (UID: \"ee36cd67-b157-4fda-bd62-34ba6cd9460e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" Nov 02 20:27:17.448249 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.448233 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee36cd67-b157-4fda-bd62-34ba6cd9460e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj\" (UID: \"ee36cd67-b157-4fda-bd62-34ba6cd9460e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" Nov 02 20:27:17.505376 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.505349 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5pqf\" (UniqueName: \"kubernetes.io/projected/ee36cd67-b157-4fda-bd62-34ba6cd9460e-kube-api-access-t5pqf\") pod \"payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj\" (UID: \"ee36cd67-b157-4fda-bd62-34ba6cd9460e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" Nov 02 20:27:17.546842 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.546785 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" Nov 02 20:27:17.697405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.697377 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" Nov 02 20:27:17.915046 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:27:17.915011 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee36cd67_b157_4fda_bd62_34ba6cd9460e.slice/crio-3200eb5efc6a00ee9ec0c1e591a648c2072aae70f10c250866c07b55c6e91a91 WatchSource:0}: Error finding container 3200eb5efc6a00ee9ec0c1e591a648c2072aae70f10c250866c07b55c6e91a91: Status 404 returned error can't find the container with id 3200eb5efc6a00ee9ec0c1e591a648c2072aae70f10c250866c07b55c6e91a91 Nov 02 20:27:17.915171 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:17.915065 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj"] Nov 02 20:27:18.068108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:18.068064 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn"] Nov 02 20:27:18.069291 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:27:18.069269 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8daff05_2927_4e7b_95a4_201024607e67.slice/crio-82aa14b06550625f7e006146031342f2d72ef1edb4370b28fc69b819adb61e23 WatchSource:0}: Error finding container 82aa14b06550625f7e006146031342f2d72ef1edb4370b28fc69b819adb61e23: Status 404 returned error can't find the container with id 82aa14b06550625f7e006146031342f2d72ef1edb4370b28fc69b819adb61e23 Nov 02 20:27:18.447402 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:18.447368 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:18.608361 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:18.608324 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" event={"ID":"e8daff05-2927-4e7b-95a4-201024607e67","Type":"ContainerStarted","Data":"0cd4f0a112f288becc144823eea65127280484fbb051bbd11ca1e8ec0aaa5b28"} Nov 02 20:27:18.608555 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:18.608391 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" event={"ID":"e8daff05-2927-4e7b-95a4-201024607e67","Type":"ContainerStarted","Data":"82aa14b06550625f7e006146031342f2d72ef1edb4370b28fc69b819adb61e23"} Nov 02 20:27:18.608555 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:18.608530 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" Nov 02 20:27:18.610154 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:18.610118 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" event={"ID":"ee36cd67-b157-4fda-bd62-34ba6cd9460e","Type":"ContainerStarted","Data":"b99d29e8e91d614e4104aab2c219fe72b407dd2273f525cd20a6e07546143efd"} Nov 02 20:27:18.610274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:18.610168 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" event={"ID":"ee36cd67-b157-4fda-bd62-34ba6cd9460e","Type":"ContainerStarted","Data":"3200eb5efc6a00ee9ec0c1e591a648c2072aae70f10c250866c07b55c6e91a91"} Nov 02 20:27:18.610274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:18.610208 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" Nov 02 20:27:18.693319 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:18.693288 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:18.717989 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:18.717914 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" podStartSLOduration=1.717902482 podStartE2EDuration="1.717902482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:18.716733499 +0000 UTC m=+767.006390028" watchObservedRunningTime="2026-11-02 20:27:18.717902482 +0000 UTC m=+767.007559389" Nov 02 20:27:18.828527 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:18.828407 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" podStartSLOduration=1.82835063 podStartE2EDuration="1.82835063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:18.825866156 +0000 UTC m=+767.115522685" watchObservedRunningTime="2026-11-02 20:27:18.82835063 +0000 UTC m=+767.118007159" Nov 02 20:27:18.944863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:18.944817 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:19.344727 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.343942 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:19.586904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.586873 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:19.654954 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.654921 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j"] Nov 02 20:27:19.658777 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.658760 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" Nov 02 20:27:19.710229 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.710207 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-bce768\"" Nov 02 20:27:19.710381 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.710288 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-bce768-dockercfg-96xcb\"" Nov 02 20:27:19.713370 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.713347 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j"] Nov 02 20:27:19.774699 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.774671 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j\" (UID: \"f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" Nov 02 20:27:19.775237 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.774889 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mc2v\" (UniqueName: \"kubernetes.io/projected/f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c-kube-api-access-7mc2v\") pod \"payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j\" (UID: \"f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" Nov 02 20:27:19.799239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.799215 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v"] Nov 02 20:27:19.803187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.803170 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" Nov 02 20:27:19.857205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.857182 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v"] Nov 02 20:27:19.876339 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.876309 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mc2v\" (UniqueName: \"kubernetes.io/projected/f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c-kube-api-access-7mc2v\") pod \"payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j\" (UID: \"f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" Nov 02 20:27:19.876447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.876372 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j\" (UID: \"f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" Nov 02 20:27:19.877324 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.877299 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j\" (UID: \"f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" Nov 02 20:27:19.935010 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.934949 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mc2v\" (UniqueName: \"kubernetes.io/projected/f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c-kube-api-access-7mc2v\") pod \"payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j\" (UID: \"f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" Nov 02 20:27:19.971754 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.971735 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" Nov 02 20:27:19.977260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.977239 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv2ft\" (UniqueName: \"kubernetes.io/projected/2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3-kube-api-access-nv2ft\") pod \"payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v\" (UID: \"2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" Nov 02 20:27:19.977347 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:19.977288 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v\" (UID: \"2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" Nov 02 20:27:20.078598 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:20.078569 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv2ft\" (UniqueName: \"kubernetes.io/projected/2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3-kube-api-access-nv2ft\") pod \"payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v\" (UID: \"2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" Nov 02 20:27:20.078718 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:20.078607 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v\" (UID: \"2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" Nov 02 20:27:20.079256 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:20.079236 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v\" (UID: \"2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" Nov 02 20:27:20.136507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:20.136483 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv2ft\" (UniqueName: \"kubernetes.io/projected/2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3-kube-api-access-nv2ft\") pod \"payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v\" (UID: \"2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" Nov 02 20:27:20.338522 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:20.338500 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j"] Nov 02 20:27:20.340366 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:27:20.340333 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3075dbf_1cf7_4f40_9b8a_a8d6423ed53c.slice/crio-adb961d880f859e4097b968508da879d9e4a4a7ed556cae240093d85ae44137a WatchSource:0}: Error finding container adb961d880f859e4097b968508da879d9e4a4a7ed556cae240093d85ae44137a: Status 404 returned error can't find the container with id adb961d880f859e4097b968508da879d9e4a4a7ed556cae240093d85ae44137a Nov 02 20:27:20.416153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:20.416129 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" Nov 02 20:27:20.621894 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:20.621861 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" event={"ID":"f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c","Type":"ContainerStarted","Data":"07b7d016cb65a3c2559cd8bcee7e975aa8da695985f1a5ee49aae19d7d18beb3"} Nov 02 20:27:20.622058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:20.621912 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" event={"ID":"f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c","Type":"ContainerStarted","Data":"adb961d880f859e4097b968508da879d9e4a4a7ed556cae240093d85ae44137a"} Nov 02 20:27:20.622058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:20.621965 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" Nov 02 20:27:20.731663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:20.731612 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" podStartSLOduration=1.731598706 podStartE2EDuration="1.731598706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:20.730353527 +0000 UTC m=+769.020010055" watchObservedRunningTime="2026-11-02 20:27:20.731598706 +0000 UTC m=+769.021255213" Nov 02 20:27:20.781710 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:20.781690 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v"] Nov 02 20:27:20.782912 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:27:20.782886 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef8c062_bd4c_4aaa_b95a_d56c1c7c68b3.slice/crio-1951a294063694d07b2d1b51ab4c516f4c944d3233f11eea3bd524d4ef178e34 WatchSource:0}: Error finding container 1951a294063694d07b2d1b51ab4c516f4c944d3233f11eea3bd524d4ef178e34: Status 404 returned error can't find the container with id 1951a294063694d07b2d1b51ab4c516f4c944d3233f11eea3bd524d4ef178e34 Nov 02 20:27:21.628671 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:21.628638 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" event={"ID":"2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3","Type":"ContainerStarted","Data":"329c36247d67f0b919b5878124ef7f872924c5ff9564961b3f0b0931a3818a43"} Nov 02 20:27:21.628835 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:21.628691 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" event={"ID":"2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3","Type":"ContainerStarted","Data":"1951a294063694d07b2d1b51ab4c516f4c944d3233f11eea3bd524d4ef178e34"} Nov 02 20:27:21.628880 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:21.628837 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" Nov 02 20:27:21.988603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:21.988205 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:22.267177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:22.267050 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:22.927760 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:22.927723 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:23.173621 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:23.173586 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:23.442489 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:23.442452 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:23.867018 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:23.866976 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:24.113898 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:24.113861 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:24.363851 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:24.363816 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:25.938589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:25.938562 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:27.015201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.015163 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:27.262618 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.262587 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:27.297726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.297631 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" podStartSLOduration=8.297617886 podStartE2EDuration="8.297617886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:21.740574795 +0000 UTC m=+770.030231322" watchObservedRunningTime="2026-11-02 20:27:27.297617886 +0000 UTC m=+775.587274413" Nov 02 20:27:27.298996 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.298978 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp"] Nov 02 20:27:27.302894 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.302876 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" Nov 02 20:27:27.357932 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.357901 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e1c40c-fa5a-4707-8d39-24e1d57b0d91-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp\" (UID: \"c4e1c40c-fa5a-4707-8d39-24e1d57b0d91\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" Nov 02 20:27:27.358113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.357942 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnfp8\" (UniqueName: \"kubernetes.io/projected/c4e1c40c-fa5a-4707-8d39-24e1d57b0d91-kube-api-access-bnfp8\") pod \"payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp\" (UID: \"c4e1c40c-fa5a-4707-8d39-24e1d57b0d91\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" Nov 02 20:27:27.401441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.401408 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-c49693\"" Nov 02 20:27:27.401926 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.401898 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-c49693-dockercfg-gxjph\"" Nov 02 20:27:27.405409 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.405387 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp"] Nov 02 20:27:27.453111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.441303 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42"] Nov 02 20:27:27.456152 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.456132 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" Nov 02 20:27:27.458787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.458767 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e1c40c-fa5a-4707-8d39-24e1d57b0d91-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp\" (UID: \"c4e1c40c-fa5a-4707-8d39-24e1d57b0d91\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" Nov 02 20:27:27.458893 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.458797 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnfp8\" (UniqueName: \"kubernetes.io/projected/c4e1c40c-fa5a-4707-8d39-24e1d57b0d91-kube-api-access-bnfp8\") pod \"payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp\" (UID: \"c4e1c40c-fa5a-4707-8d39-24e1d57b0d91\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" Nov 02 20:27:27.459551 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.459534 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e1c40c-fa5a-4707-8d39-24e1d57b0d91-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp\" (UID: \"c4e1c40c-fa5a-4707-8d39-24e1d57b0d91\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" Nov 02 20:27:27.493032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.493006 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42"] Nov 02 20:27:27.517075 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.517051 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnfp8\" (UniqueName: \"kubernetes.io/projected/c4e1c40c-fa5a-4707-8d39-24e1d57b0d91-kube-api-access-bnfp8\") pod \"payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp\" (UID: \"c4e1c40c-fa5a-4707-8d39-24e1d57b0d91\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" Nov 02 20:27:27.559563 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.559496 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p55lw\" (UniqueName: \"kubernetes.io/projected/96107d44-a6a7-4af1-840b-706c09bf59a1-kube-api-access-p55lw\") pod \"payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42\" (UID: \"96107d44-a6a7-4af1-840b-706c09bf59a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" Nov 02 20:27:27.559563 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.559546 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96107d44-a6a7-4af1-840b-706c09bf59a1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42\" (UID: \"96107d44-a6a7-4af1-840b-706c09bf59a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" Nov 02 20:27:27.604299 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.604271 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" Nov 02 20:27:27.616804 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.616778 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" Nov 02 20:27:27.660985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.660954 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p55lw\" (UniqueName: \"kubernetes.io/projected/96107d44-a6a7-4af1-840b-706c09bf59a1-kube-api-access-p55lw\") pod \"payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42\" (UID: \"96107d44-a6a7-4af1-840b-706c09bf59a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" Nov 02 20:27:27.661154 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.661023 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96107d44-a6a7-4af1-840b-706c09bf59a1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42\" (UID: \"96107d44-a6a7-4af1-840b-706c09bf59a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" Nov 02 20:27:27.661950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.661921 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96107d44-a6a7-4af1-840b-706c09bf59a1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42\" (UID: \"96107d44-a6a7-4af1-840b-706c09bf59a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" Nov 02 20:27:27.753655 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.753618 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p55lw\" (UniqueName: \"kubernetes.io/projected/96107d44-a6a7-4af1-840b-706c09bf59a1-kube-api-access-p55lw\") pod \"payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42\" (UID: \"96107d44-a6a7-4af1-840b-706c09bf59a1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" Nov 02 20:27:27.770248 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:27.770213 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" Nov 02 20:27:28.015528 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:28.015504 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp"] Nov 02 20:27:28.017207 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:27:28.017177 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e1c40c_fa5a_4707_8d39_24e1d57b0d91.slice/crio-fc257d7ac591deaa64f2ef82de1a48647d5fe6ea1b8be6b9f18ff6c7c5823c37 WatchSource:0}: Error finding container fc257d7ac591deaa64f2ef82de1a48647d5fe6ea1b8be6b9f18ff6c7c5823c37: Status 404 returned error can't find the container with id fc257d7ac591deaa64f2ef82de1a48647d5fe6ea1b8be6b9f18ff6c7c5823c37 Nov 02 20:27:28.166038 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:28.166011 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42"] Nov 02 20:27:28.167517 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:27:28.167489 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96107d44_a6a7_4af1_840b_706c09bf59a1.slice/crio-82d58eee8f3c3dc18fea2690c9c10003a9e309de7110e141c209307719b1608e WatchSource:0}: Error finding container 82d58eee8f3c3dc18fea2690c9c10003a9e309de7110e141c209307719b1608e: Status 404 returned error can't find the container with id 82d58eee8f3c3dc18fea2690c9c10003a9e309de7110e141c209307719b1608e Nov 02 20:27:28.681300 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:28.681262 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" event={"ID":"c4e1c40c-fa5a-4707-8d39-24e1d57b0d91","Type":"ContainerStarted","Data":"2ba174f8e1d3678c5a27e4fc7a531d5ecfc0d151e194eecd4a69d18c95e21b19"} Nov 02 20:27:28.681478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:28.681316 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" event={"ID":"c4e1c40c-fa5a-4707-8d39-24e1d57b0d91","Type":"ContainerStarted","Data":"fc257d7ac591deaa64f2ef82de1a48647d5fe6ea1b8be6b9f18ff6c7c5823c37"} Nov 02 20:27:28.681478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:28.681376 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" Nov 02 20:27:28.682987 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:28.682965 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" event={"ID":"96107d44-a6a7-4af1-840b-706c09bf59a1","Type":"ContainerStarted","Data":"ee735e00b0479fb65c40adfe1f0b9778dcebaeb39ba74337739d6e1288e4dfe7"} Nov 02 20:27:28.683120 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:28.682998 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" event={"ID":"96107d44-a6a7-4af1-840b-706c09bf59a1","Type":"ContainerStarted","Data":"82d58eee8f3c3dc18fea2690c9c10003a9e309de7110e141c209307719b1608e"} Nov 02 20:27:28.683120 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:28.683071 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" Nov 02 20:27:28.805034 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:28.804972 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" podStartSLOduration=1.804954544 podStartE2EDuration="1.804954544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:28.804281231 +0000 UTC m=+777.093937761" watchObservedRunningTime="2026-11-02 20:27:28.804954544 +0000 UTC m=+777.094611073" Nov 02 20:27:28.916735 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:28.916648 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" podStartSLOduration=1.916629449 podStartE2EDuration="1.916629449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:28.914064926 +0000 UTC m=+777.203721454" watchObservedRunningTime="2026-11-02 20:27:28.916629449 +0000 UTC m=+777.206285977" Nov 02 20:27:29.299688 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:29.299654 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:29.547156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:29.545688 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:29.618182 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:29.618157 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" Nov 02 20:27:29.618339 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:29.618222 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" Nov 02 20:27:31.648171 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:31.648143 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" Nov 02 20:27:32.001898 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:32.001811 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:32.248512 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:32.248481 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:32.296493 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:32.296425 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7"] Nov 02 20:27:32.302335 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:32.302314 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" Nov 02 20:27:32.352292 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:32.352267 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7"] Nov 02 20:27:32.352505 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:32.352329 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-694cee\"" Nov 02 20:27:32.352505 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:32.352445 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-694cee-dockercfg-dwfzr\"" Nov 02 20:27:32.408410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:32.408357 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7\" (UID: \"7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" Nov 02 20:27:32.408723 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:32.408442 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc2hl\" (UniqueName: \"kubernetes.io/projected/7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e-kube-api-access-kc2hl\") pod \"payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7\" (UID: \"7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" Nov 02 20:27:32.509369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:32.509333 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7\" (UID: \"7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" Nov 02 20:27:32.509552 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:32.509396 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc2hl\" (UniqueName: \"kubernetes.io/projected/7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e-kube-api-access-kc2hl\") pod \"payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7\" (UID: \"7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" Nov 02 20:27:32.510441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:32.510417 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7\" (UID: \"7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" Nov 02 20:27:32.567650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:32.567584 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc2hl\" (UniqueName: \"kubernetes.io/projected/7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e-kube-api-access-kc2hl\") pod \"payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7\" (UID: \"7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" Nov 02 20:27:32.616215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:32.616186 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" Nov 02 20:27:32.640386 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:32.640364 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" Nov 02 20:27:32.984479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:32.984441 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7"] Nov 02 20:27:33.344262 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:27:33.344177 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:27:33.714169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:33.714135 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" event={"ID":"7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e","Type":"ContainerStarted","Data":"36ad480acfe3810928acd5502e2c3d2a0506f7451a2610ee5fb02856923ecf43"} Nov 02 20:27:33.714360 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:33.714188 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" event={"ID":"7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e","Type":"ContainerStarted","Data":"e672bfcd87cd1d22938d23c76f1668e65a1c93b42a2021fab5f0c756a958ea98"} Nov 02 20:27:33.714360 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:33.714252 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" Nov 02 20:27:33.834015 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:33.833945 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" podStartSLOduration=1.833924364 podStartE2EDuration="1.833924364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:33.823901883 +0000 UTC m=+782.113558412" watchObservedRunningTime="2026-11-02 20:27:33.833924364 +0000 UTC m=+782.123580893" Nov 02 20:27:34.683592 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:34.683517 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:34.930637 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:34.930604 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:35.569219 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:35.569189 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:27:35.570560 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:35.569251 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" podUID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:27:35.727799 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:35.727764 4971 generic.go:356] "Generic (PLEG): container finished" podID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" containerID="ce529c7e4d15bf92455208c281ff7dcf850d819a17919501a8153c17fc9ab454" exitCode=0 Nov 02 20:27:35.728184 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:35.727838 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" event={"ID":"41baf9c8-0dc6-4e97-958c-70391cbeb98b","Type":"ContainerDied","Data":"ce529c7e4d15bf92455208c281ff7dcf850d819a17919501a8153c17fc9ab454"} Nov 02 20:27:35.728254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:35.728233 4971 scope.go:122] "RemoveContainer" containerID="ce529c7e4d15bf92455208c281ff7dcf850d819a17919501a8153c17fc9ab454" Nov 02 20:27:35.729523 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:35.729495 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:35.729586 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:35.729568 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:35.729629 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:35.729600 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:36.753301 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:36.753266 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" event={"ID":"41baf9c8-0dc6-4e97-958c-70391cbeb98b","Type":"ContainerStarted","Data":"4f37038f58f86692833fddc244ef59f7a885c316c8a8526af2065da734fda6be"} Nov 02 20:27:37.370889 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:37.370857 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:37.371108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:37.371070 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 02 20:27:37.371163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:37.371134 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" podUID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 02 20:27:38.370623 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:38.370590 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 02 20:27:38.371019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:38.370657 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" podUID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 02 20:27:39.371326 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.371299 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 02 20:27:39.371763 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.371736 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" podUID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 02 20:27:39.448069 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.448002 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:39.692559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.692479 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" Nov 02 20:27:39.692559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.692560 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" Nov 02 20:27:39.702430 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.702400 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:39.792995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.792964 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558"] Nov 02 20:27:39.799062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.799046 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" Nov 02 20:27:39.817449 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.817424 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558"] Nov 02 20:27:39.850273 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.850128 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-2250b4\"" Nov 02 20:27:39.850636 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.850620 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-2250b4-dockercfg-j4pvv\"" Nov 02 20:27:39.883132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.883111 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79b7b\" (UniqueName: \"kubernetes.io/projected/ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48-kube-api-access-79b7b\") pod \"payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558\" (UID: \"ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" Nov 02 20:27:39.883381 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.883366 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558\" (UID: \"ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" Nov 02 20:27:39.900599 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.900579 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t"] Nov 02 20:27:39.904601 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.904585 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" Nov 02 20:27:39.986596 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.986511 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t"] Nov 02 20:27:40.000121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.994283 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79b7b\" (UniqueName: \"kubernetes.io/projected/ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48-kube-api-access-79b7b\") pod \"payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558\" (UID: \"ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" Nov 02 20:27:40.000375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.000348 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29a43c7-6192-4151-9c21-bcedab45ba8c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t\" (UID: \"a29a43c7-6192-4151-9c21-bcedab45ba8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" Nov 02 20:27:40.000515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.000498 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwb6t\" (UniqueName: \"kubernetes.io/projected/a29a43c7-6192-4151-9c21-bcedab45ba8c-kube-api-access-cwb6t\") pod \"payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t\" (UID: \"a29a43c7-6192-4151-9c21-bcedab45ba8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" Nov 02 20:27:40.000601 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:39.998706 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:40.000652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.000608 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558\" (UID: \"ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" Nov 02 20:27:40.001406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.001382 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558\" (UID: \"ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" Nov 02 20:27:40.057346 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.057311 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79b7b\" (UniqueName: \"kubernetes.io/projected/ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48-kube-api-access-79b7b\") pod \"payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558\" (UID: \"ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" Nov 02 20:27:40.101454 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.101416 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwb6t\" (UniqueName: \"kubernetes.io/projected/a29a43c7-6192-4151-9c21-bcedab45ba8c-kube-api-access-cwb6t\") pod \"payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t\" (UID: \"a29a43c7-6192-4151-9c21-bcedab45ba8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" Nov 02 20:27:40.101593 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.101548 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29a43c7-6192-4151-9c21-bcedab45ba8c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t\" (UID: \"a29a43c7-6192-4151-9c21-bcedab45ba8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" Nov 02 20:27:40.102211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.102193 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29a43c7-6192-4151-9c21-bcedab45ba8c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t\" (UID: \"a29a43c7-6192-4151-9c21-bcedab45ba8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" Nov 02 20:27:40.113278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.113258 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" Nov 02 20:27:40.159680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.159657 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwb6t\" (UniqueName: \"kubernetes.io/projected/a29a43c7-6192-4151-9c21-bcedab45ba8c-kube-api-access-cwb6t\") pod \"payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t\" (UID: \"a29a43c7-6192-4151-9c21-bcedab45ba8c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" Nov 02 20:27:40.219029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.219001 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" Nov 02 20:27:40.370879 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.370845 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 02 20:27:40.371023 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.370909 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" podUID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 02 20:27:40.508761 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.508731 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558"] Nov 02 20:27:40.509527 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:27:40.509500 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad50ecfc_027a_4ec8_adf7_f3bd20ae4c48.slice/crio-623f63bd49d7535e012e6423b6c137163e40b7f8ad62837ca4d998860907616e WatchSource:0}: Error finding container 623f63bd49d7535e012e6423b6c137163e40b7f8ad62837ca4d998860907616e: Status 404 returned error can't find the container with id 623f63bd49d7535e012e6423b6c137163e40b7f8ad62837ca4d998860907616e Nov 02 20:27:40.613809 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.613785 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t"] Nov 02 20:27:40.614501 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:27:40.614475 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29a43c7_6192_4151_9c21_bcedab45ba8c.slice/crio-41ca70ad9f3624bfd43f9c243ce1e6781ded7f7b11e6924f8b64f21c8efb4afc WatchSource:0}: Error finding container 41ca70ad9f3624bfd43f9c243ce1e6781ded7f7b11e6924f8b64f21c8efb4afc: Status 404 returned error can't find the container with id 41ca70ad9f3624bfd43f9c243ce1e6781ded7f7b11e6924f8b64f21c8efb4afc Nov 02 20:27:40.776447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.776363 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" event={"ID":"ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48","Type":"ContainerStarted","Data":"da848c22a288dc7a52c88904ff7c4c9dc75a9b6b7556a00424b59c76eaa85466"} Nov 02 20:27:40.776447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.776420 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" event={"ID":"ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48","Type":"ContainerStarted","Data":"623f63bd49d7535e012e6423b6c137163e40b7f8ad62837ca4d998860907616e"} Nov 02 20:27:40.776625 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.776561 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" Nov 02 20:27:40.778264 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.778230 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" event={"ID":"a29a43c7-6192-4151-9c21-bcedab45ba8c","Type":"ContainerStarted","Data":"47065991283d41332d9dd31f44e4252ec6be2c9b4c15f51df5d8f9d2ef3da751"} Nov 02 20:27:40.778370 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.778276 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" Nov 02 20:27:40.778370 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.778291 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" event={"ID":"a29a43c7-6192-4151-9c21-bcedab45ba8c","Type":"ContainerStarted","Data":"41ca70ad9f3624bfd43f9c243ce1e6781ded7f7b11e6924f8b64f21c8efb4afc"} Nov 02 20:27:40.904707 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:40.904650 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" podStartSLOduration=1.904637529 podStartE2EDuration="1.904637529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:40.902546143 +0000 UTC m=+789.192202683" watchObservedRunningTime="2026-11-02 20:27:40.904637529 +0000 UTC m=+789.194294056" Nov 02 20:27:41.014828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:41.014449 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" podStartSLOduration=2.014430359 podStartE2EDuration="2.014430359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:41.012510028 +0000 UTC m=+789.302166557" watchObservedRunningTime="2026-11-02 20:27:41.014430359 +0000 UTC m=+789.304086889" Nov 02 20:27:41.370980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:41.370946 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 02 20:27:41.371199 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:41.370997 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" podUID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 02 20:27:42.374942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:42.374913 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:42.375424 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:42.375382 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:42.376148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:42.376130 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:27:44.555412 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:44.555363 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:44.724545 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:44.724514 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" Nov 02 20:27:49.414269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:49.414228 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:51.787801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:51.787770 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" Nov 02 20:27:51.788223 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:51.788110 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" Nov 02 20:27:54.072833 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:54.072802 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:54.323801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:54.323711 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:54.926168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:54.926096 4971 generic.go:356] "Generic (PLEG): container finished" podID="0e0eeb03-8f43-4579-bdb9-66e80f676abb" containerID="c6bce39071dea22b90e40e0a91e3a1cd7bf1f4142c20aad00c5a8b11a40924b8" exitCode=0 Nov 02 20:27:54.926168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:54.926110 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" event={"ID":"0e0eeb03-8f43-4579-bdb9-66e80f676abb","Type":"ContainerDied","Data":"c6bce39071dea22b90e40e0a91e3a1cd7bf1f4142c20aad00c5a8b11a40924b8"} Nov 02 20:27:54.926482 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:54.926467 4971 scope.go:122] "RemoveContainer" containerID="c6bce39071dea22b90e40e0a91e3a1cd7bf1f4142c20aad00c5a8b11a40924b8" Nov 02 20:27:54.927919 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:54.927885 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:54.928000 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:54.927982 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:54.928062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:54.928044 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:55.950850 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:55.950814 4971 generic.go:356] "Generic (PLEG): container finished" podID="413b0cf5-5c2e-4983-8437-7f9d2e899405" containerID="327b455ca520a38bbc20f7043c200b3fc50952ad9e43941dacd42e699668857e" exitCode=0 Nov 02 20:27:55.951382 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:55.950887 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" event={"ID":"413b0cf5-5c2e-4983-8437-7f9d2e899405","Type":"ContainerDied","Data":"327b455ca520a38bbc20f7043c200b3fc50952ad9e43941dacd42e699668857e"} Nov 02 20:27:55.951382 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:55.951320 4971 scope.go:122] "RemoveContainer" containerID="327b455ca520a38bbc20f7043c200b3fc50952ad9e43941dacd42e699668857e" Nov 02 20:27:55.952854 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:55.952815 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:55.952958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:55.952900 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:55.952958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:55.952942 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:27:55.953034 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:55.952996 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" event={"ID":"0e0eeb03-8f43-4579-bdb9-66e80f676abb","Type":"ContainerStarted","Data":"fa66e63489b4c9d3d31e5c98f10c91e5c87822e3722d84f6f68bc5e829c0af40"} Nov 02 20:27:56.289428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:56.289333 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:27:56.289585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:56.289508 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 02 20:27:56.289585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:56.289575 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" podUID="0e0eeb03-8f43-4579-bdb9-66e80f676abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 02 20:27:56.977070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:56.977029 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" event={"ID":"413b0cf5-5c2e-4983-8437-7f9d2e899405","Type":"ContainerStarted","Data":"adfcb49e6fabffbe0ae0d2556837a3c35acc2d91de68e0c02187fdcf85162583"} Nov 02 20:27:57.290337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:57.290252 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 02 20:27:57.290486 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:57.290342 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" podUID="0e0eeb03-8f43-4579-bdb9-66e80f676abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 02 20:27:57.359870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:57.359824 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:57.602664 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:57.602631 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:27:57.602863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:57.602764 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 02 20:27:57.602863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:57.602807 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" podUID="413b0cf5-5c2e-4983-8437-7f9d2e899405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 02 20:27:57.956162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:57.955291 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:58.290294 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:58.290209 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 02 20:27:58.290661 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:58.290307 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" podUID="0e0eeb03-8f43-4579-bdb9-66e80f676abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 02 20:27:58.334109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:58.331906 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:27:58.603385 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:58.603362 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 02 20:27:58.603542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:58.603414 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" podUID="413b0cf5-5c2e-4983-8437-7f9d2e899405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 02 20:27:59.290329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:59.290290 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 02 20:27:59.290745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:59.290344 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" podUID="0e0eeb03-8f43-4579-bdb9-66e80f676abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 02 20:27:59.603566 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:59.603535 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 02 20:27:59.603797 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:59.603771 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" podUID="413b0cf5-5c2e-4983-8437-7f9d2e899405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 02 20:27:59.717653 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:27:59.717609 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:00.290156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:00.290121 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 02 20:28:00.290360 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:00.290174 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" podUID="0e0eeb03-8f43-4579-bdb9-66e80f676abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 02 20:28:00.603197 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:00.603162 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 02 20:28:00.603361 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:00.603217 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" podUID="413b0cf5-5c2e-4983-8437-7f9d2e899405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 02 20:28:01.293477 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:01.293451 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:28:01.293894 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:01.293705 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:28:01.294384 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:01.294361 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:28:01.603456 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:01.603426 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 02 20:28:01.603640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:01.603487 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" podUID="413b0cf5-5c2e-4983-8437-7f9d2e899405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 02 20:28:01.899280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:01.899193 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:02.619413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:02.619378 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:28:02.621268 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:02.621245 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:28:02.624418 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:02.624384 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:28:03.337814 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:28:03.337777 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:28:04.871878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:04.871844 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:08.354558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:08.354517 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:08.993514 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:08.993481 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:09.243368 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:09.243300 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:09.944778 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:09.944738 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:13.608033 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:13.608003 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:14.171058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:14.170986 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:14.422950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:14.422853 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:15.194225 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:15.194192 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:18.643592 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:18.643558 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:19.290551 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:19.290520 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:19.536314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:19.536277 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:20.499758 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:20.499725 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:21.909105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:21.909052 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2"] Nov 02 20:28:21.913070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:21.913052 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:21.967713 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:21.967686 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2"] Nov 02 20:28:22.011259 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.011223 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bce768\"/\"openshift-service-ca.crt\"" Nov 02 20:28:22.011406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.011310 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bce768\"/\"kube-root-ca.crt\"" Nov 02 20:28:22.011406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.011332 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bce768\"/\"simulator-w4-bce768-kserve-self-signed-certs\"" Nov 02 20:28:22.011406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.011332 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bce768\"/\"default-dockercfg-jb4t8\"" Nov 02 20:28:22.031009 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.030981 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5kz2\" (UniqueName: \"kubernetes.io/projected/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-kube-api-access-t5kz2\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.031158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.031141 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-tls-certs\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.031219 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.031169 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-tmp-dir\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.031219 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.031191 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-model-cache\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.031364 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.031343 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-dshm\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.031425 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.031412 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-home\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.113161 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.113129 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:22.132490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.132463 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-dshm\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.132640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.132511 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-home\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.132640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.132536 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5kz2\" (UniqueName: \"kubernetes.io/projected/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-kube-api-access-t5kz2\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.132640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.132600 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-tls-certs\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.132640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.132619 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-tmp-dir\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.132640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.132638 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-model-cache\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.132993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.132964 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-home\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.133150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.133005 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-model-cache\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.133150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.133029 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-tmp-dir\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.134965 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.134945 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-dshm\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.135137 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.135121 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-tls-certs\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.190699 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.190637 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5kz2\" (UniqueName: \"kubernetes.io/projected/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-kube-api-access-t5kz2\") pod \"simulator-w4-bce768-kserve-669b8d5956-jv4w2\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.225528 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.225500 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:22.595495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.595468 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2"] Nov 02 20:28:22.596574 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:28:22.596540 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bf3f8ce_7c86_4fa5_ac99_bf2532808882.slice/crio-6d061230adb7d558d87c3738caca1dfe78721e9c0896018eee65f399fdc0544d WatchSource:0}: Error finding container 6d061230adb7d558d87c3738caca1dfe78721e9c0896018eee65f399fdc0544d: Status 404 returned error can't find the container with id 6d061230adb7d558d87c3738caca1dfe78721e9c0896018eee65f399fdc0544d Nov 02 20:28:22.986349 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:22.986316 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:23.166837 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:23.166806 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" event={"ID":"9bf3f8ce-7c86-4fa5-ac99-bf2532808882","Type":"ContainerStarted","Data":"9383f2d99c9f96b84b23fb1f07e8956306e14b07c7b692470327c2a9230e409e"} Nov 02 20:28:23.166837 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:23.166842 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" event={"ID":"9bf3f8ce-7c86-4fa5-ac99-bf2532808882","Type":"ContainerStarted","Data":"6d061230adb7d558d87c3738caca1dfe78721e9c0896018eee65f399fdc0544d"} Nov 02 20:28:23.225971 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:23.225947 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:23.280437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:23.280346 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podStartSLOduration=2.280333718 podStartE2EDuration="2.280333718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:23.278715817 +0000 UTC m=+831.568372345" watchObservedRunningTime="2026-11-02 20:28:23.280333718 +0000 UTC m=+831.569990248" Nov 02 20:28:23.695596 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:23.695563 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:23.945728 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:23.945648 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:25.851595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:25.851555 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:26.298703 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:26.298620 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:26.544534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:26.544493 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:30.917579 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:30.917550 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:32.225989 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.225960 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:32.238400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.238377 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:32.339419 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.339393 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t"] Nov 02 20:28:32.343334 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.343314 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.353488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.353466 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t"] Nov 02 20:28:32.392817 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.392790 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-61cdc1\"/\"kube-root-ca.crt\"" Nov 02 20:28:32.392817 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.392814 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-61cdc1\"/\"openshift-service-ca.crt\"" Nov 02 20:28:32.392991 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.392933 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-61cdc1\"/\"default-dockercfg-jl9z8\"" Nov 02 20:28:32.393038 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.393025 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-61cdc1\"/\"simulator-w3-61cdc1-kserve-self-signed-certs\"" Nov 02 20:28:32.431148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.431072 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-model-cache\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.431304 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.431163 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-tls-certs\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.431304 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.431224 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-home\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.431304 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.431247 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-dshm\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.431304 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.431280 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfcs2\" (UniqueName: \"kubernetes.io/projected/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-kube-api-access-gfcs2\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.431622 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.431325 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-tmp-dir\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.487234 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.487145 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:32.531887 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.531863 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-model-cache\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.532034 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.531896 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-tls-certs\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.532034 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.531929 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-home\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.532034 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.531951 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-dshm\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.532034 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.531985 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfcs2\" (UniqueName: \"kubernetes.io/projected/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-kube-api-access-gfcs2\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.532034 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.532024 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-tmp-dir\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.532356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.532334 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-model-cache\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.532396 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.532374 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-home\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.532457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.532445 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-tmp-dir\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.534435 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.534415 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-dshm\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.534822 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.534803 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-tls-certs\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.591498 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.591471 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfcs2\" (UniqueName: \"kubernetes.io/projected/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-kube-api-access-gfcs2\") pod \"simulator-w3-61cdc1-kserve-955c8b8-gdb8t\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:32.656951 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:32.656924 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:33.027819 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:33.027792 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t"] Nov 02 20:28:33.028745 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:28:33.028706 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa6f0ad6_43f0_484a_9f00_55f7dcef5698.slice/crio-998ea4add1ed641c8479edff91e46ae93f896605af32a0c7e84190b6e4afb7e3 WatchSource:0}: Error finding container 998ea4add1ed641c8479edff91e46ae93f896605af32a0c7e84190b6e4afb7e3: Status 404 returned error can't find the container with id 998ea4add1ed641c8479edff91e46ae93f896605af32a0c7e84190b6e4afb7e3 Nov 02 20:28:33.224166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:33.224063 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" event={"ID":"aa6f0ad6-43f0-484a-9f00-55f7dcef5698","Type":"ContainerStarted","Data":"d2607e7731514643c5be9110331e89d06781e9f7f4595bde90be8e0d7307a008"} Nov 02 20:28:33.224166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:33.224128 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" event={"ID":"aa6f0ad6-43f0-484a-9f00-55f7dcef5698","Type":"ContainerStarted","Data":"998ea4add1ed641c8479edff91e46ae93f896605af32a0c7e84190b6e4afb7e3"} Nov 02 20:28:33.235276 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:33.235250 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:28:33.336983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:33.336876 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podStartSLOduration=1.336861852 podStartE2EDuration="1.336861852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:33.334609623 +0000 UTC m=+841.624266148" watchObservedRunningTime="2026-11-02 20:28:33.336861852 +0000 UTC m=+841.626518433" Nov 02 20:28:33.339425 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:28:33.339375 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:28:33.657967 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:33.657936 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:35.730780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:35.730746 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:39.270179 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:39.270148 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:39.912279 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:39.912220 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:40.166138 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:40.166053 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:40.618530 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:40.618498 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:42.657381 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:42.657301 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:42.669993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:42.669968 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:43.040501 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:43.040433 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:43.298655 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:43.298574 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:28:43.353998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:43.353963 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:43.648669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:43.648628 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:43.892057 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:43.892022 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:45.719902 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:45.719869 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:50.515671 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:50.515641 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:51.755875 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:51.755841 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:55.316051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:55.316020 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:28:56.934534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:28:56.934498 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:00.116447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:00.116417 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:03.338039 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:29:03.338001 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:29:04.922871 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:04.922835 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:07.006746 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.006712 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7"] Nov 02 20:29:07.011838 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.011814 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.062798 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.062770 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7"] Nov 02 20:29:07.094400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.094374 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-model-cache\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.094525 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.094417 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lkw8\" (UniqueName: \"kubernetes.io/projected/b1b99b04-38f8-4120-a237-8fc9e60071ed-kube-api-access-8lkw8\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.094595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.094530 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-dshm\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.094595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.094550 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-tmp-dir\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.094702 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.094675 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1b99b04-38f8-4120-a237-8fc9e60071ed-tls-certs\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.094793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.094778 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-home\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.106651 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.106628 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-694cee\"/\"openshift-service-ca.crt\"" Nov 02 20:29:07.106747 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.106694 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-694cee\"/\"kube-root-ca.crt\"" Nov 02 20:29:07.108145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.108123 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-694cee\"/\"default-dockercfg-sdqsf\"" Nov 02 20:29:07.108271 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.108158 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-694cee\"/\"simulator-w2-694cee-kserve-self-signed-certs\"" Nov 02 20:29:07.196275 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.196247 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-home\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.196457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.196284 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-model-cache\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.196457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.196324 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lkw8\" (UniqueName: \"kubernetes.io/projected/b1b99b04-38f8-4120-a237-8fc9e60071ed-kube-api-access-8lkw8\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.196457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.196368 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-dshm\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.196457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.196394 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-tmp-dir\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.196690 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.196474 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1b99b04-38f8-4120-a237-8fc9e60071ed-tls-certs\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.196750 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.196725 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-home\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.196810 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.196788 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-model-cache\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.196879 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.196855 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-tmp-dir\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.198793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.198777 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-dshm\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.199058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.199041 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1b99b04-38f8-4120-a237-8fc9e60071ed-tls-certs\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.218665 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.218643 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:07.256038 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.256016 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lkw8\" (UniqueName: \"kubernetes.io/projected/b1b99b04-38f8-4120-a237-8fc9e60071ed-kube-api-access-8lkw8\") pod \"simulator-w2-694cee-kserve-78b55d74dc-5gcn7\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.324364 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.324300 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:07.696722 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:07.696680 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7"] Nov 02 20:29:07.697986 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:29:07.697939 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1b99b04_38f8_4120_a237_8fc9e60071ed.slice/crio-3f2101d8c9e37396ab66f05dcfcc0699d25953893c6d6aa29801a122604299e8 WatchSource:0}: Error finding container 3f2101d8c9e37396ab66f05dcfcc0699d25953893c6d6aa29801a122604299e8: Status 404 returned error can't find the container with id 3f2101d8c9e37396ab66f05dcfcc0699d25953893c6d6aa29801a122604299e8 Nov 02 20:29:08.427513 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:08.427478 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" event={"ID":"b1b99b04-38f8-4120-a237-8fc9e60071ed","Type":"ContainerStarted","Data":"6c17b3380b325eb376b5178e3ad199262306235c04697777d3f91370e7194bb9"} Nov 02 20:29:08.427513 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:08.427524 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" event={"ID":"b1b99b04-38f8-4120-a237-8fc9e60071ed","Type":"ContainerStarted","Data":"3f2101d8c9e37396ab66f05dcfcc0699d25953893c6d6aa29801a122604299e8"} Nov 02 20:29:08.539662 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:08.539613 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podStartSLOduration=2.5395996739999998 podStartE2EDuration="2.539599674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:08.538035067 +0000 UTC m=+876.827691596" watchObservedRunningTime="2026-11-02 20:29:08.539599674 +0000 UTC m=+876.829256198" Nov 02 20:29:09.325153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:09.325122 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:09.808788 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:09.808751 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:12.254565 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:12.254528 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:14.706313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:14.706280 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:17.325293 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:17.325253 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:17.337410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:17.337382 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:17.492679 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:17.492648 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:29:19.414196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:19.414160 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:23.330163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:23.330102 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:24.228522 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:24.228482 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:25.943741 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:25.943703 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:26.183647 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:26.183604 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:26.635767 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:26.635733 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:28.983907 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:28.983869 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:31.947031 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:31.947000 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:29:31.952006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:31.948787 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:29:33.010915 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:33.010879 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:33.338985 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:29:33.338904 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:29:33.821845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:33.821815 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:38.609152 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:38.609114 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:43.385252 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:43.385217 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:43.935514 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:43.935464 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j"] Nov 02 20:29:43.939571 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:43.939553 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:43.993876 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:43.993844 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j"] Nov 02 20:29:44.037152 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.037130 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-61cdc1\"/\"premium-w3-61cdc1-kserve-self-signed-certs\"" Nov 02 20:29:44.068553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.068526 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-model-cache\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.068948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.068892 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht92t\" (UniqueName: \"kubernetes.io/projected/4486f93b-6513-4998-a2c7-d55f05805ee1-kube-api-access-ht92t\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.070955 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.070931 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-home\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.071245 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.071224 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4486f93b-6513-4998-a2c7-d55f05805ee1-tls-certs\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.071435 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.071419 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-tmp-dir\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.071699 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.071666 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-dshm\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.137289 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.137242 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:44.172536 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.172507 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-home\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.172683 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.172583 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4486f93b-6513-4998-a2c7-d55f05805ee1-tls-certs\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.172683 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.172624 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-tmp-dir\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.172786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.172718 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-dshm\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.172786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.172747 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-model-cache\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.172892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.172807 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht92t\" (UniqueName: \"kubernetes.io/projected/4486f93b-6513-4998-a2c7-d55f05805ee1-kube-api-access-ht92t\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.172956 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.172927 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-home\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.173073 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.173053 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-tmp-dir\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.173199 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.173180 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-model-cache\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.175052 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.175028 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-dshm\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.175353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.175335 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4486f93b-6513-4998-a2c7-d55f05805ee1-tls-certs\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.232787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.232724 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht92t\" (UniqueName: \"kubernetes.io/projected/4486f93b-6513-4998-a2c7-d55f05805ee1-kube-api-access-ht92t\") pod \"premium-w3-61cdc1-kserve-5f6757cdcc-jww2j\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.251965 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.251943 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:44.641215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:44.641190 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j"] Nov 02 20:29:44.642706 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:29:44.642663 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4486f93b_6513_4998_a2c7_d55f05805ee1.slice/crio-c515302eae4abe714b256e759efc94e3673c95191dda76bc9c726d1253bd74ff WatchSource:0}: Error finding container c515302eae4abe714b256e759efc94e3673c95191dda76bc9c726d1253bd74ff: Status 404 returned error can't find the container with id c515302eae4abe714b256e759efc94e3673c95191dda76bc9c726d1253bd74ff Nov 02 20:29:45.652956 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:45.652915 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" event={"ID":"4486f93b-6513-4998-a2c7-d55f05805ee1","Type":"ContainerStarted","Data":"cde583b432e41744fe7e87c7dfbd2f4ee8c394fe5b25f89d4e0e5b16d6026534"} Nov 02 20:29:45.652956 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:45.652958 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" event={"ID":"4486f93b-6513-4998-a2c7-d55f05805ee1","Type":"ContainerStarted","Data":"c515302eae4abe714b256e759efc94e3673c95191dda76bc9c726d1253bd74ff"} Nov 02 20:29:45.764256 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:45.764211 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podStartSLOduration=2.764197169 podStartE2EDuration="2.764197169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:45.763993831 +0000 UTC m=+914.053650359" watchObservedRunningTime="2026-11-02 20:29:45.764197169 +0000 UTC m=+914.053853737" Nov 02 20:29:46.252998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:46.252967 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:48.255529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:48.255496 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:53.113945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:53.113890 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:29:54.253025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:54.252988 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:54.265751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:54.265728 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:54.719613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:54.719582 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:29:57.917078 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:29:57.917006 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:00.068684 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.068644 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:00.171895 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.171854 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp"] Nov 02 20:30:00.175989 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.175960 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f"] Nov 02 20:30:00.176146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.176095 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" Nov 02 20:30:00.179657 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.179639 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk"] Nov 02 20:30:00.179785 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.179769 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" Nov 02 20:30:00.183005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.182986 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" Nov 02 20:30:00.220020 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.219997 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd"] Nov 02 20:30:00.244970 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.237833 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/853568b4-1289-42dc-aaa8-1f0654101bea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk\" (UID: \"853568b4-1289-42dc-aaa8-1f0654101bea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" Nov 02 20:30:00.245322 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.245293 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52ebc70-c0d2-440a-951b-741be403aced-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894190-45dkp\" (UID: \"c52ebc70-c0d2-440a-951b-741be403aced\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" Nov 02 20:30:00.245508 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.245489 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z58zx\" (UniqueName: \"kubernetes.io/projected/853568b4-1289-42dc-aaa8-1f0654101bea-kube-api-access-z58zx\") pod \"maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk\" (UID: \"853568b4-1289-42dc-aaa8-1f0654101bea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" Nov 02 20:30:00.245645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.245626 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvrjj\" (UniqueName: \"kubernetes.io/projected/0f2e0cfc-b518-4daa-933f-10cb82d72184-kube-api-access-qvrjj\") pod \"maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f\" (UID: \"0f2e0cfc-b518-4daa-933f-10cb82d72184\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" Nov 02 20:30:00.245839 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.245820 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mwxl\" (UniqueName: \"kubernetes.io/projected/c52ebc70-c0d2-440a-951b-741be403aced-kube-api-access-2mwxl\") pod \"maas-api-key-cleanup-29894190-45dkp\" (UID: \"c52ebc70-c0d2-440a-951b-741be403aced\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" Nov 02 20:30:00.246067 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.246048 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2e0cfc-b518-4daa-933f-10cb82d72184-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f\" (UID: \"0f2e0cfc-b518-4daa-933f-10cb82d72184\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" Nov 02 20:30:00.246331 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.242220 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s"] Nov 02 20:30:00.258242 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.242320 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" Nov 02 20:30:00.269915 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.263532 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:30:00.278774 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.278750 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x"] Nov 02 20:30:00.284125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.283511 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" Nov 02 20:30:00.285720 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.285697 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp"] Nov 02 20:30:00.285912 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.285743 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f"] Nov 02 20:30:00.285912 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.285772 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk"] Nov 02 20:30:00.285912 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.285853 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" Nov 02 20:30:00.287431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.287409 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd"] Nov 02 20:30:00.290062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.290041 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s"] Nov 02 20:30:00.342763 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.342707 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x"] Nov 02 20:30:00.347655 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.347633 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c239e090-9178-4ab1-a71d-dd83391df358-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s\" (UID: \"c239e090-9178-4ab1-a71d-dd83391df358\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" Nov 02 20:30:00.347764 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.347697 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2e0cfc-b518-4daa-933f-10cb82d72184-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f\" (UID: \"0f2e0cfc-b518-4daa-933f-10cb82d72184\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" Nov 02 20:30:00.347832 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.347771 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c68zm\" (UniqueName: \"kubernetes.io/projected/8a48b92a-6991-485e-8d41-de59766de14e-kube-api-access-c68zm\") pod \"maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x\" (UID: \"8a48b92a-6991-485e-8d41-de59766de14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" Nov 02 20:30:00.347832 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.347818 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/853568b4-1289-42dc-aaa8-1f0654101bea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk\" (UID: \"853568b4-1289-42dc-aaa8-1f0654101bea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" Nov 02 20:30:00.347947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.347852 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh46x\" (UniqueName: \"kubernetes.io/projected/fd4e27eb-15d7-48c0-8612-c605fac7fe38-kube-api-access-qh46x\") pod \"maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd\" (UID: \"fd4e27eb-15d7-48c0-8612-c605fac7fe38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" Nov 02 20:30:00.348498 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.348477 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/853568b4-1289-42dc-aaa8-1f0654101bea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk\" (UID: \"853568b4-1289-42dc-aaa8-1f0654101bea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" Nov 02 20:30:00.348594 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.348527 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xbx9\" (UniqueName: \"kubernetes.io/projected/c239e090-9178-4ab1-a71d-dd83391df358-kube-api-access-7xbx9\") pod \"maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s\" (UID: \"c239e090-9178-4ab1-a71d-dd83391df358\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" Nov 02 20:30:00.348594 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.348567 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2e0cfc-b518-4daa-933f-10cb82d72184-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f\" (UID: \"0f2e0cfc-b518-4daa-933f-10cb82d72184\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" Nov 02 20:30:00.348594 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.348579 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd4e27eb-15d7-48c0-8612-c605fac7fe38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd\" (UID: \"fd4e27eb-15d7-48c0-8612-c605fac7fe38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" Nov 02 20:30:00.348712 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.348692 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52ebc70-c0d2-440a-951b-741be403aced-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894190-45dkp\" (UID: \"c52ebc70-c0d2-440a-951b-741be403aced\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" Nov 02 20:30:00.348755 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.348732 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a48b92a-6991-485e-8d41-de59766de14e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x\" (UID: \"8a48b92a-6991-485e-8d41-de59766de14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" Nov 02 20:30:00.348832 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.348817 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z58zx\" (UniqueName: \"kubernetes.io/projected/853568b4-1289-42dc-aaa8-1f0654101bea-kube-api-access-z58zx\") pod \"maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk\" (UID: \"853568b4-1289-42dc-aaa8-1f0654101bea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" Nov 02 20:30:00.348876 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.348851 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvrjj\" (UniqueName: \"kubernetes.io/projected/0f2e0cfc-b518-4daa-933f-10cb82d72184-kube-api-access-qvrjj\") pod \"maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f\" (UID: \"0f2e0cfc-b518-4daa-933f-10cb82d72184\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" Nov 02 20:30:00.348946 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.348930 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mwxl\" (UniqueName: \"kubernetes.io/projected/c52ebc70-c0d2-440a-951b-741be403aced-kube-api-access-2mwxl\") pod \"maas-api-key-cleanup-29894190-45dkp\" (UID: \"c52ebc70-c0d2-440a-951b-741be403aced\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" Nov 02 20:30:00.349387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.349369 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52ebc70-c0d2-440a-951b-741be403aced-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894190-45dkp\" (UID: \"c52ebc70-c0d2-440a-951b-741be403aced\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" Nov 02 20:30:00.409071 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.409015 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mwxl\" (UniqueName: \"kubernetes.io/projected/c52ebc70-c0d2-440a-951b-741be403aced-kube-api-access-2mwxl\") pod \"maas-api-key-cleanup-29894190-45dkp\" (UID: \"c52ebc70-c0d2-440a-951b-741be403aced\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" Nov 02 20:30:00.412234 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.412216 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvrjj\" (UniqueName: \"kubernetes.io/projected/0f2e0cfc-b518-4daa-933f-10cb82d72184-kube-api-access-qvrjj\") pod \"maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f\" (UID: \"0f2e0cfc-b518-4daa-933f-10cb82d72184\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" Nov 02 20:30:00.412596 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.412579 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z58zx\" (UniqueName: \"kubernetes.io/projected/853568b4-1289-42dc-aaa8-1f0654101bea-kube-api-access-z58zx\") pod \"maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk\" (UID: \"853568b4-1289-42dc-aaa8-1f0654101bea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" Nov 02 20:30:00.450070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.450048 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a48b92a-6991-485e-8d41-de59766de14e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x\" (UID: \"8a48b92a-6991-485e-8d41-de59766de14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" Nov 02 20:30:00.450226 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.450211 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c239e090-9178-4ab1-a71d-dd83391df358-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s\" (UID: \"c239e090-9178-4ab1-a71d-dd83391df358\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" Nov 02 20:30:00.450282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.450270 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c68zm\" (UniqueName: \"kubernetes.io/projected/8a48b92a-6991-485e-8d41-de59766de14e-kube-api-access-c68zm\") pod \"maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x\" (UID: \"8a48b92a-6991-485e-8d41-de59766de14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" Nov 02 20:30:00.450322 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.450301 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh46x\" (UniqueName: \"kubernetes.io/projected/fd4e27eb-15d7-48c0-8612-c605fac7fe38-kube-api-access-qh46x\") pod \"maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd\" (UID: \"fd4e27eb-15d7-48c0-8612-c605fac7fe38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" Nov 02 20:30:00.450355 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.450321 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xbx9\" (UniqueName: \"kubernetes.io/projected/c239e090-9178-4ab1-a71d-dd83391df358-kube-api-access-7xbx9\") pod \"maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s\" (UID: \"c239e090-9178-4ab1-a71d-dd83391df358\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" Nov 02 20:30:00.450355 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.450344 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd4e27eb-15d7-48c0-8612-c605fac7fe38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd\" (UID: \"fd4e27eb-15d7-48c0-8612-c605fac7fe38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" Nov 02 20:30:00.450661 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.450633 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a48b92a-6991-485e-8d41-de59766de14e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x\" (UID: \"8a48b92a-6991-485e-8d41-de59766de14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" Nov 02 20:30:00.450897 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.450878 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c239e090-9178-4ab1-a71d-dd83391df358-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s\" (UID: \"c239e090-9178-4ab1-a71d-dd83391df358\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" Nov 02 20:30:00.451011 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.450994 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd4e27eb-15d7-48c0-8612-c605fac7fe38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd\" (UID: \"fd4e27eb-15d7-48c0-8612-c605fac7fe38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" Nov 02 20:30:00.492505 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.492483 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" Nov 02 20:30:00.499892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.499876 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" Nov 02 20:30:00.505758 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.505733 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" Nov 02 20:30:00.509216 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.509191 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c68zm\" (UniqueName: \"kubernetes.io/projected/8a48b92a-6991-485e-8d41-de59766de14e-kube-api-access-c68zm\") pod \"maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x\" (UID: \"8a48b92a-6991-485e-8d41-de59766de14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" Nov 02 20:30:00.509519 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.509499 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xbx9\" (UniqueName: \"kubernetes.io/projected/c239e090-9178-4ab1-a71d-dd83391df358-kube-api-access-7xbx9\") pod \"maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s\" (UID: \"c239e090-9178-4ab1-a71d-dd83391df358\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" Nov 02 20:30:00.510237 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.510213 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh46x\" (UniqueName: \"kubernetes.io/projected/fd4e27eb-15d7-48c0-8612-c605fac7fe38-kube-api-access-qh46x\") pod \"maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd\" (UID: \"fd4e27eb-15d7-48c0-8612-c605fac7fe38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" Nov 02 20:30:00.586191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.586165 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" Nov 02 20:30:00.605601 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.605368 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" Nov 02 20:30:00.612508 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.611306 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" Nov 02 20:30:00.875425 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.875395 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp"] Nov 02 20:30:00.876008 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:30:00.875981 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52ebc70_c0d2_440a_951b_741be403aced.slice/crio-de112b124261846c3d4a1cc92cf902ce94beb513c2ecba0aace7bb6c837eebd1 WatchSource:0}: Error finding container de112b124261846c3d4a1cc92cf902ce94beb513c2ecba0aace7bb6c837eebd1: Status 404 returned error can't find the container with id de112b124261846c3d4a1cc92cf902ce94beb513c2ecba0aace7bb6c837eebd1 Nov 02 20:30:00.921881 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:30:00.921853 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2e0cfc_b518_4daa_933f_10cb82d72184.slice/crio-9d2d918626732d8f919eb678570aeb4a36ddcd1f3071cb13b2f43a41dc118e20 WatchSource:0}: Error finding container 9d2d918626732d8f919eb678570aeb4a36ddcd1f3071cb13b2f43a41dc118e20: Status 404 returned error can't find the container with id 9d2d918626732d8f919eb678570aeb4a36ddcd1f3071cb13b2f43a41dc118e20 Nov 02 20:30:00.922292 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.922269 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f"] Nov 02 20:30:00.924715 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:30:00.924695 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853568b4_1289_42dc_aaa8_1f0654101bea.slice/crio-18159782e9093b3d611fe5fa418fb0deddf419dbb92928cd20be443d7c2ed333 WatchSource:0}: Error finding container 18159782e9093b3d611fe5fa418fb0deddf419dbb92928cd20be443d7c2ed333: Status 404 returned error can't find the container with id 18159782e9093b3d611fe5fa418fb0deddf419dbb92928cd20be443d7c2ed333 Nov 02 20:30:00.925384 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.925323 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk"] Nov 02 20:30:00.988117 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.988071 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd"] Nov 02 20:30:00.989589 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:30:00.989547 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4e27eb_15d7_48c0_8612_c605fac7fe38.slice/crio-3894270354d136501b5b8a0c331ea38924fcd4d0d2501d313f34267f895c9660 WatchSource:0}: Error finding container 3894270354d136501b5b8a0c331ea38924fcd4d0d2501d313f34267f895c9660: Status 404 returned error can't find the container with id 3894270354d136501b5b8a0c331ea38924fcd4d0d2501d313f34267f895c9660 Nov 02 20:30:00.991366 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:30:00.991341 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc239e090_9178_4ab1_a71d_dd83391df358.slice/crio-be86298b9014a8053b2a8f4e617f395aaa9b25abbd13446592cf79d8a7a9277e WatchSource:0}: Error finding container be86298b9014a8053b2a8f4e617f395aaa9b25abbd13446592cf79d8a7a9277e: Status 404 returned error can't find the container with id be86298b9014a8053b2a8f4e617f395aaa9b25abbd13446592cf79d8a7a9277e Nov 02 20:30:00.992169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.992137 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s"] Nov 02 20:30:00.994040 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:30:00.994019 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a48b92a_6991_485e_8d41_de59766de14e.slice/crio-2153ddd388f1a1908219275b43cef4e571eb13b100ec30f313cc7d5a14e21fce WatchSource:0}: Error finding container 2153ddd388f1a1908219275b43cef4e571eb13b100ec30f313cc7d5a14e21fce: Status 404 returned error can't find the container with id 2153ddd388f1a1908219275b43cef4e571eb13b100ec30f313cc7d5a14e21fce Nov 02 20:30:00.995001 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:00.994976 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x"] Nov 02 20:30:01.750821 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:01.750789 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" event={"ID":"fd4e27eb-15d7-48c0-8612-c605fac7fe38","Type":"ContainerStarted","Data":"3894270354d136501b5b8a0c331ea38924fcd4d0d2501d313f34267f895c9660"} Nov 02 20:30:01.752098 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:01.752051 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" event={"ID":"853568b4-1289-42dc-aaa8-1f0654101bea","Type":"ContainerStarted","Data":"18159782e9093b3d611fe5fa418fb0deddf419dbb92928cd20be443d7c2ed333"} Nov 02 20:30:01.753346 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:01.753322 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" event={"ID":"c52ebc70-c0d2-440a-951b-741be403aced","Type":"ContainerStarted","Data":"de112b124261846c3d4a1cc92cf902ce94beb513c2ecba0aace7bb6c837eebd1"} Nov 02 20:30:01.754655 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:01.754632 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" event={"ID":"0f2e0cfc-b518-4daa-933f-10cb82d72184","Type":"ContainerStarted","Data":"9d2d918626732d8f919eb678570aeb4a36ddcd1f3071cb13b2f43a41dc118e20"} Nov 02 20:30:01.755851 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:01.755823 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" event={"ID":"c239e090-9178-4ab1-a71d-dd83391df358","Type":"ContainerStarted","Data":"be86298b9014a8053b2a8f4e617f395aaa9b25abbd13446592cf79d8a7a9277e"} Nov 02 20:30:01.756998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:01.756976 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" event={"ID":"8a48b92a-6991-485e-8d41-de59766de14e","Type":"ContainerStarted","Data":"2153ddd388f1a1908219275b43cef4e571eb13b100ec30f313cc7d5a14e21fce"} Nov 02 20:30:02.725525 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:02.725489 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:03.338426 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:03.338388 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:30:07.613064 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:07.612939 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:07.811736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:07.811372 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" event={"ID":"853568b4-1289-42dc-aaa8-1f0654101bea","Type":"ContainerStarted","Data":"9e1ca95eeba1b3035c82612df171bff3619c400a3e532b1efcdb5719cf9283f6"} Nov 02 20:30:07.921695 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:07.921633 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" podStartSLOduration=1.741224532 podStartE2EDuration="7.921614512s" totalImagesPullingTime="6.18038998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:07.920434995 +0000 UTC m=+936.210091524" watchObservedRunningTime="2026-11-02 20:30:07.921614512 +0000 UTC m=+936.211271042" Nov 02 20:30:08.820424 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:08.820388 4971 generic.go:356] "Generic (PLEG): container finished" podID="8a48b92a-6991-485e-8d41-de59766de14e" containerID="fbbf1f4029e271ada076bfb43f8ccee8ba91e868337b10100c0f218cb52b9d4e" exitCode=60 Nov 02 20:30:08.820857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:08.820463 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" event={"ID":"8a48b92a-6991-485e-8d41-de59766de14e","Type":"ContainerDied","Data":"fbbf1f4029e271ada076bfb43f8ccee8ba91e868337b10100c0f218cb52b9d4e"} Nov 02 20:30:08.820857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:08.820733 4971 scope.go:122] "RemoveContainer" containerID="fbbf1f4029e271ada076bfb43f8ccee8ba91e868337b10100c0f218cb52b9d4e" Nov 02 20:30:08.837795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:08.823065 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" event={"ID":"fd4e27eb-15d7-48c0-8612-c605fac7fe38","Type":"ContainerStarted","Data":"56f78716c2367a8867601000ea86f9667e304402ed6113d3c460f659dd04f4ee"} Nov 02 20:30:08.837795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:08.829019 4971 generic.go:356] "Generic (PLEG): container finished" podID="853568b4-1289-42dc-aaa8-1f0654101bea" containerID="9e1ca95eeba1b3035c82612df171bff3619c400a3e532b1efcdb5719cf9283f6" exitCode=60 Nov 02 20:30:08.837795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:08.829152 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" event={"ID":"853568b4-1289-42dc-aaa8-1f0654101bea","Type":"ContainerDied","Data":"9e1ca95eeba1b3035c82612df171bff3619c400a3e532b1efcdb5719cf9283f6"} Nov 02 20:30:08.837795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:08.829483 4971 scope.go:122] "RemoveContainer" containerID="9e1ca95eeba1b3035c82612df171bff3619c400a3e532b1efcdb5719cf9283f6" Nov 02 20:30:08.837795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:08.832056 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" event={"ID":"c52ebc70-c0d2-440a-951b-741be403aced","Type":"ContainerStarted","Data":"26f13a056024ad1f80274e161fdd9f6436457a71ff26d294bb51ddb4f9a8110c"} Nov 02 20:30:08.837795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:08.836724 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" event={"ID":"0f2e0cfc-b518-4daa-933f-10cb82d72184","Type":"ContainerStarted","Data":"065e5dda3f1fb8141025df9dd234ba4f30a9118aa826cebbeb9884b0d8c6ff18"} Nov 02 20:30:09.149668 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.149599 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" podStartSLOduration=2.301004218 podStartE2EDuration="9.149584278s" totalImagesPullingTime="6.84858006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:09.148513306 +0000 UTC m=+937.438169833" watchObservedRunningTime="2026-11-02 20:30:09.149584278 +0000 UTC m=+937.439240799" Nov 02 20:30:09.256695 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.256632 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" podStartSLOduration=2.12409073 podStartE2EDuration="9.256619704s" totalImagesPullingTime="7.132528974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:09.256421502 +0000 UTC m=+937.546078030" watchObservedRunningTime="2026-11-02 20:30:09.256619704 +0000 UTC m=+937.546276232" Nov 02 20:30:09.366565 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.366517 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" podStartSLOduration=2.00838629 podStartE2EDuration="9.366502475s" totalImagesPullingTime="7.358116185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:09.364731019 +0000 UTC m=+937.654387548" watchObservedRunningTime="2026-11-02 20:30:09.366502475 +0000 UTC m=+937.656159008" Nov 02 20:30:09.845408 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.845380 4971 generic.go:356] "Generic (PLEG): container finished" podID="853568b4-1289-42dc-aaa8-1f0654101bea" containerID="36fae56df1a388ddd588e76b30b91d7ab491b8f38e2fa467d0a8a5f994768491" exitCode=60 Nov 02 20:30:09.845849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.845465 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" event={"ID":"853568b4-1289-42dc-aaa8-1f0654101bea","Type":"ContainerDied","Data":"36fae56df1a388ddd588e76b30b91d7ab491b8f38e2fa467d0a8a5f994768491"} Nov 02 20:30:09.845849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.845527 4971 scope.go:122] "RemoveContainer" containerID="9e1ca95eeba1b3035c82612df171bff3619c400a3e532b1efcdb5719cf9283f6" Nov 02 20:30:09.845849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.845738 4971 scope.go:122] "RemoveContainer" containerID="36fae56df1a388ddd588e76b30b91d7ab491b8f38e2fa467d0a8a5f994768491" Nov 02 20:30:09.846059 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:09.846038 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk_odh-ai-gateway-infra(853568b4-1289-42dc-aaa8-1f0654101bea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" podUID="853568b4-1289-42dc-aaa8-1f0654101bea" Nov 02 20:30:09.847363 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.847338 4971 generic.go:356] "Generic (PLEG): container finished" podID="c52ebc70-c0d2-440a-951b-741be403aced" containerID="26f13a056024ad1f80274e161fdd9f6436457a71ff26d294bb51ddb4f9a8110c" exitCode=60 Nov 02 20:30:09.847452 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.847416 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" event={"ID":"c52ebc70-c0d2-440a-951b-741be403aced","Type":"ContainerDied","Data":"26f13a056024ad1f80274e161fdd9f6436457a71ff26d294bb51ddb4f9a8110c"} Nov 02 20:30:09.847640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.847627 4971 scope.go:122] "RemoveContainer" containerID="26f13a056024ad1f80274e161fdd9f6436457a71ff26d294bb51ddb4f9a8110c" Nov 02 20:30:09.848986 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.848962 4971 generic.go:356] "Generic (PLEG): container finished" podID="0f2e0cfc-b518-4daa-933f-10cb82d72184" containerID="065e5dda3f1fb8141025df9dd234ba4f30a9118aa826cebbeb9884b0d8c6ff18" exitCode=60 Nov 02 20:30:09.849145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.849127 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" event={"ID":"0f2e0cfc-b518-4daa-933f-10cb82d72184","Type":"ContainerDied","Data":"065e5dda3f1fb8141025df9dd234ba4f30a9118aa826cebbeb9884b0d8c6ff18"} Nov 02 20:30:09.849249 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.849227 4971 scope.go:122] "RemoveContainer" containerID="065e5dda3f1fb8141025df9dd234ba4f30a9118aa826cebbeb9884b0d8c6ff18" Nov 02 20:30:09.851588 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.851568 4971 generic.go:356] "Generic (PLEG): container finished" podID="c239e090-9178-4ab1-a71d-dd83391df358" containerID="d0eef9c56eadf521113514d262a4999b48fad6b8c5bbd1bd658f2377d35fc69e" exitCode=60 Nov 02 20:30:09.851716 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.851696 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" event={"ID":"c239e090-9178-4ab1-a71d-dd83391df358","Type":"ContainerDied","Data":"d0eef9c56eadf521113514d262a4999b48fad6b8c5bbd1bd658f2377d35fc69e"} Nov 02 20:30:09.851913 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.851891 4971 scope.go:122] "RemoveContainer" containerID="d0eef9c56eadf521113514d262a4999b48fad6b8c5bbd1bd658f2377d35fc69e" Nov 02 20:30:09.853927 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.853910 4971 generic.go:356] "Generic (PLEG): container finished" podID="8a48b92a-6991-485e-8d41-de59766de14e" containerID="310ea0b11e3c1c95d6cfd7714f50367f7f0503ac0d90aab6657e3da01ca442f8" exitCode=60 Nov 02 20:30:09.854005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.853987 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" event={"ID":"8a48b92a-6991-485e-8d41-de59766de14e","Type":"ContainerDied","Data":"310ea0b11e3c1c95d6cfd7714f50367f7f0503ac0d90aab6657e3da01ca442f8"} Nov 02 20:30:09.854245 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.854228 4971 scope.go:122] "RemoveContainer" containerID="310ea0b11e3c1c95d6cfd7714f50367f7f0503ac0d90aab6657e3da01ca442f8" Nov 02 20:30:09.854474 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:09.854424 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x_odh-ai-gateway-infra(8a48b92a-6991-485e-8d41-de59766de14e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" podUID="8a48b92a-6991-485e-8d41-de59766de14e" Nov 02 20:30:09.855649 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.855630 4971 generic.go:356] "Generic (PLEG): container finished" podID="fd4e27eb-15d7-48c0-8612-c605fac7fe38" containerID="56f78716c2367a8867601000ea86f9667e304402ed6113d3c460f659dd04f4ee" exitCode=60 Nov 02 20:30:09.855735 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.855667 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" event={"ID":"fd4e27eb-15d7-48c0-8612-c605fac7fe38","Type":"ContainerDied","Data":"56f78716c2367a8867601000ea86f9667e304402ed6113d3c460f659dd04f4ee"} Nov 02 20:30:09.855851 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.855838 4971 scope.go:122] "RemoveContainer" containerID="56f78716c2367a8867601000ea86f9667e304402ed6113d3c460f659dd04f4ee" Nov 02 20:30:09.864525 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:09.864502 4971 scope.go:122] "RemoveContainer" containerID="fbbf1f4029e271ada076bfb43f8ccee8ba91e868337b10100c0f218cb52b9d4e" Nov 02 20:30:10.864161 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.864134 4971 generic.go:356] "Generic (PLEG): container finished" podID="c239e090-9178-4ab1-a71d-dd83391df358" containerID="45fe220b550c195dc865d94ed23269a266d477cfc462e94eb2e7c4b545cb4214" exitCode=60 Nov 02 20:30:10.864503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.864217 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" event={"ID":"c239e090-9178-4ab1-a71d-dd83391df358","Type":"ContainerDied","Data":"45fe220b550c195dc865d94ed23269a266d477cfc462e94eb2e7c4b545cb4214"} Nov 02 20:30:10.864503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.864315 4971 scope.go:122] "RemoveContainer" containerID="d0eef9c56eadf521113514d262a4999b48fad6b8c5bbd1bd658f2377d35fc69e" Nov 02 20:30:10.864631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.864534 4971 scope.go:122] "RemoveContainer" containerID="45fe220b550c195dc865d94ed23269a266d477cfc462e94eb2e7c4b545cb4214" Nov 02 20:30:10.864848 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:10.864821 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s_odh-ai-gateway-infra(c239e090-9178-4ab1-a71d-dd83391df358)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" podUID="c239e090-9178-4ab1-a71d-dd83391df358" Nov 02 20:30:10.866220 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.866196 4971 scope.go:122] "RemoveContainer" containerID="310ea0b11e3c1c95d6cfd7714f50367f7f0503ac0d90aab6657e3da01ca442f8" Nov 02 20:30:10.866496 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:10.866466 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x_odh-ai-gateway-infra(8a48b92a-6991-485e-8d41-de59766de14e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" podUID="8a48b92a-6991-485e-8d41-de59766de14e" Nov 02 20:30:10.867828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.867811 4971 generic.go:356] "Generic (PLEG): container finished" podID="fd4e27eb-15d7-48c0-8612-c605fac7fe38" containerID="e5b96ae9b6e2e33528b39bd573410fd1edff0213f8fc6f9ad47a752b53d70481" exitCode=60 Nov 02 20:30:10.867910 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.867887 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" event={"ID":"fd4e27eb-15d7-48c0-8612-c605fac7fe38","Type":"ContainerDied","Data":"e5b96ae9b6e2e33528b39bd573410fd1edff0213f8fc6f9ad47a752b53d70481"} Nov 02 20:30:10.868128 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.868107 4971 scope.go:122] "RemoveContainer" containerID="e5b96ae9b6e2e33528b39bd573410fd1edff0213f8fc6f9ad47a752b53d70481" Nov 02 20:30:10.868341 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:10.868320 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd_odh-ai-gateway-infra(fd4e27eb-15d7-48c0-8612-c605fac7fe38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" podUID="fd4e27eb-15d7-48c0-8612-c605fac7fe38" Nov 02 20:30:10.869689 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.869671 4971 scope.go:122] "RemoveContainer" containerID="36fae56df1a388ddd588e76b30b91d7ab491b8f38e2fa467d0a8a5f994768491" Nov 02 20:30:10.869875 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:10.869856 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk_odh-ai-gateway-infra(853568b4-1289-42dc-aaa8-1f0654101bea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" podUID="853568b4-1289-42dc-aaa8-1f0654101bea" Nov 02 20:30:10.871406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.871388 4971 generic.go:356] "Generic (PLEG): container finished" podID="c52ebc70-c0d2-440a-951b-741be403aced" containerID="17eab485dcf248f125a6b80a65d0f64ca29dbd2dca969837395aa19f5f8c1438" exitCode=60 Nov 02 20:30:10.871512 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.871468 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" event={"ID":"c52ebc70-c0d2-440a-951b-741be403aced","Type":"ContainerDied","Data":"17eab485dcf248f125a6b80a65d0f64ca29dbd2dca969837395aa19f5f8c1438"} Nov 02 20:30:10.871642 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.871626 4971 scope.go:122] "RemoveContainer" containerID="17eab485dcf248f125a6b80a65d0f64ca29dbd2dca969837395aa19f5f8c1438" Nov 02 20:30:10.871842 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:10.871823 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894190-45dkp_odh-ai-gateway-infra(c52ebc70-c0d2-440a-951b-741be403aced)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" podUID="c52ebc70-c0d2-440a-951b-741be403aced" Nov 02 20:30:10.873238 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.873219 4971 generic.go:356] "Generic (PLEG): container finished" podID="0f2e0cfc-b518-4daa-933f-10cb82d72184" containerID="dde75b1d594e6a959f5979c6b4155a948ba6e0eaae82e8989e9dc6447bf5d6c7" exitCode=60 Nov 02 20:30:10.873317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.873283 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" event={"ID":"0f2e0cfc-b518-4daa-933f-10cb82d72184","Type":"ContainerDied","Data":"dde75b1d594e6a959f5979c6b4155a948ba6e0eaae82e8989e9dc6447bf5d6c7"} Nov 02 20:30:10.873483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.873471 4971 scope.go:122] "RemoveContainer" containerID="dde75b1d594e6a959f5979c6b4155a948ba6e0eaae82e8989e9dc6447bf5d6c7" Nov 02 20:30:10.873650 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:10.873635 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f_odh-ai-gateway-infra(0f2e0cfc-b518-4daa-933f-10cb82d72184)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" podUID="0f2e0cfc-b518-4daa-933f-10cb82d72184" Nov 02 20:30:10.940715 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.940700 4971 scope.go:122] "RemoveContainer" containerID="56f78716c2367a8867601000ea86f9667e304402ed6113d3c460f659dd04f4ee" Nov 02 20:30:10.952865 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.952847 4971 scope.go:122] "RemoveContainer" containerID="26f13a056024ad1f80274e161fdd9f6436457a71ff26d294bb51ddb4f9a8110c" Nov 02 20:30:10.965054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:10.965037 4971 scope.go:122] "RemoveContainer" containerID="065e5dda3f1fb8141025df9dd234ba4f30a9118aa826cebbeb9884b0d8c6ff18" Nov 02 20:30:11.880740 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:11.880712 4971 scope.go:122] "RemoveContainer" containerID="17eab485dcf248f125a6b80a65d0f64ca29dbd2dca969837395aa19f5f8c1438" Nov 02 20:30:11.881199 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:11.880946 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894190-45dkp_odh-ai-gateway-infra(c52ebc70-c0d2-440a-951b-741be403aced)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" podUID="c52ebc70-c0d2-440a-951b-741be403aced" Nov 02 20:30:11.882278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:11.882259 4971 scope.go:122] "RemoveContainer" containerID="dde75b1d594e6a959f5979c6b4155a948ba6e0eaae82e8989e9dc6447bf5d6c7" Nov 02 20:30:11.882477 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:11.882456 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f_odh-ai-gateway-infra(0f2e0cfc-b518-4daa-933f-10cb82d72184)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" podUID="0f2e0cfc-b518-4daa-933f-10cb82d72184" Nov 02 20:30:11.883781 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:11.883764 4971 scope.go:122] "RemoveContainer" containerID="45fe220b550c195dc865d94ed23269a266d477cfc462e94eb2e7c4b545cb4214" Nov 02 20:30:11.883936 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:11.883918 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s_odh-ai-gateway-infra(c239e090-9178-4ab1-a71d-dd83391df358)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" podUID="c239e090-9178-4ab1-a71d-dd83391df358" Nov 02 20:30:11.885301 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:11.885284 4971 scope.go:122] "RemoveContainer" containerID="e5b96ae9b6e2e33528b39bd573410fd1edff0213f8fc6f9ad47a752b53d70481" Nov 02 20:30:11.885471 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:11.885453 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd_odh-ai-gateway-infra(fd4e27eb-15d7-48c0-8612-c605fac7fe38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" podUID="fd4e27eb-15d7-48c0-8612-c605fac7fe38" Nov 02 20:30:12.436010 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:12.435962 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:17.140208 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.140178 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb"] Nov 02 20:30:17.144183 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.144164 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.232638 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.232604 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:17.242353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.242315 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-30ee89\"/\"openshift-service-ca.crt\"" Nov 02 20:30:17.242353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.242315 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-30ee89\"/\"premium-w0-30ee89-kserve-self-signed-certs\"" Nov 02 20:30:17.242558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.242332 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-30ee89\"/\"default-dockercfg-7plks\"" Nov 02 20:30:17.242558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.242370 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-30ee89\"/\"kube-root-ca.crt\"" Nov 02 20:30:17.245534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.245508 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb"] Nov 02 20:30:17.246009 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.245985 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrcnd\" (UniqueName: \"kubernetes.io/projected/1e563127-e308-4d93-ba3a-46efea5469dc-kube-api-access-rrcnd\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.246095 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.246031 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-dshm\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.246233 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.246187 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-home\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.246337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.246319 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-model-cache\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.246411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.246399 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e563127-e308-4d93-ba3a-46efea5469dc-tls-certs\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.246462 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.246425 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-tmp-dir\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.347174 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.347141 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-home\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.347352 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.347249 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-model-cache\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.347352 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.347329 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e563127-e308-4d93-ba3a-46efea5469dc-tls-certs\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.347457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.347353 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-tmp-dir\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.347457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.347392 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrcnd\" (UniqueName: \"kubernetes.io/projected/1e563127-e308-4d93-ba3a-46efea5469dc-kube-api-access-rrcnd\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.347457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.347423 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-dshm\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.347589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.347563 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-home\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.347623 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.347609 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-model-cache\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.347857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.347835 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-tmp-dir\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.350141 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.350071 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-dshm\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.350306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.350247 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e563127-e308-4d93-ba3a-46efea5469dc-tls-certs\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.405388 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.405323 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrcnd\" (UniqueName: \"kubernetes.io/projected/1e563127-e308-4d93-ba3a-46efea5469dc-kube-api-access-rrcnd\") pod \"premium-w0-30ee89-kserve-75d769457b-64xcb\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.456480 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.456452 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:17.484864 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.484830 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:17.845511 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.845474 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb"] Nov 02 20:30:17.845741 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:30:17.845713 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e563127_e308_4d93_ba3a_46efea5469dc.slice/crio-e0170eb7d182c9eb1ece194fa3a90a521ec3312f3b5a525f9c6f0d2f5c76304e WatchSource:0}: Error finding container e0170eb7d182c9eb1ece194fa3a90a521ec3312f3b5a525f9c6f0d2f5c76304e: Status 404 returned error can't find the container with id e0170eb7d182c9eb1ece194fa3a90a521ec3312f3b5a525f9c6f0d2f5c76304e Nov 02 20:30:17.926065 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:17.926040 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" event={"ID":"1e563127-e308-4d93-ba3a-46efea5469dc","Type":"ContainerStarted","Data":"e0170eb7d182c9eb1ece194fa3a90a521ec3312f3b5a525f9c6f0d2f5c76304e"} Nov 02 20:30:18.037831 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:18.037776 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podStartSLOduration=1.03776324 podStartE2EDuration="1.03776324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:18.036590256 +0000 UTC m=+946.326246785" watchObservedRunningTime="2026-11-02 20:30:18.03776324 +0000 UTC m=+946.327419768" Nov 02 20:30:18.457441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:18.457411 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:18.934937 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:18.934900 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" event={"ID":"1e563127-e308-4d93-ba3a-46efea5469dc","Type":"ContainerStarted","Data":"69862c9a561a26a343e043387988201f1889fd6d9a188d3d0eea58d9d35dc866"} Nov 02 20:30:21.863312 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:21.863268 4971 scope.go:122] "RemoveContainer" containerID="e5b96ae9b6e2e33528b39bd573410fd1edff0213f8fc6f9ad47a752b53d70481" Nov 02 20:30:21.863312 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:21.863306 4971 scope.go:122] "RemoveContainer" containerID="36fae56df1a388ddd588e76b30b91d7ab491b8f38e2fa467d0a8a5f994768491" Nov 02 20:30:21.863887 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:21.863873 4971 scope.go:122] "RemoveContainer" containerID="17eab485dcf248f125a6b80a65d0f64ca29dbd2dca969837395aa19f5f8c1438" Nov 02 20:30:22.022714 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:22.022545 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:22.960275 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:22.960249 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" event={"ID":"fd4e27eb-15d7-48c0-8612-c605fac7fe38","Type":"ContainerStarted","Data":"cf0e905caf6b8f080ea89a3b15179ee7a8fd6725248ac002b2e7046fa6458ae1"} Nov 02 20:30:22.962185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:22.962162 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" event={"ID":"853568b4-1289-42dc-aaa8-1f0654101bea","Type":"ContainerStarted","Data":"4710e23580fb47baa9439421d8f7cf4dbb1eae9a79dee7532a53953fd08247b3"} Nov 02 20:30:22.964018 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:22.963996 4971 generic.go:356] "Generic (PLEG): container finished" podID="c52ebc70-c0d2-440a-951b-741be403aced" containerID="7562dc4183356849ebc2a856b24e89eb5d0091a7bf77a8d8ff8549f648d5fafa" exitCode=60 Nov 02 20:30:22.964120 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:22.964069 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" event={"ID":"c52ebc70-c0d2-440a-951b-741be403aced","Type":"ContainerDied","Data":"7562dc4183356849ebc2a856b24e89eb5d0091a7bf77a8d8ff8549f648d5fafa"} Nov 02 20:30:22.964174 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:22.964148 4971 scope.go:122] "RemoveContainer" containerID="17eab485dcf248f125a6b80a65d0f64ca29dbd2dca969837395aa19f5f8c1438" Nov 02 20:30:22.964443 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:22.964423 4971 scope.go:122] "RemoveContainer" containerID="7562dc4183356849ebc2a856b24e89eb5d0091a7bf77a8d8ff8549f648d5fafa" Nov 02 20:30:22.964660 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:22.964638 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894190-45dkp_odh-ai-gateway-infra(c52ebc70-c0d2-440a-951b-741be403aced)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" podUID="c52ebc70-c0d2-440a-951b-741be403aced" Nov 02 20:30:23.859687 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:23.859659 4971 scope.go:122] "RemoveContainer" containerID="310ea0b11e3c1c95d6cfd7714f50367f7f0503ac0d90aab6657e3da01ca442f8" Nov 02 20:30:23.972998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:23.972972 4971 generic.go:356] "Generic (PLEG): container finished" podID="fd4e27eb-15d7-48c0-8612-c605fac7fe38" containerID="cf0e905caf6b8f080ea89a3b15179ee7a8fd6725248ac002b2e7046fa6458ae1" exitCode=60 Nov 02 20:30:23.973395 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:23.973046 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" event={"ID":"fd4e27eb-15d7-48c0-8612-c605fac7fe38","Type":"ContainerDied","Data":"cf0e905caf6b8f080ea89a3b15179ee7a8fd6725248ac002b2e7046fa6458ae1"} Nov 02 20:30:23.973395 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:23.973113 4971 scope.go:122] "RemoveContainer" containerID="e5b96ae9b6e2e33528b39bd573410fd1edff0213f8fc6f9ad47a752b53d70481" Nov 02 20:30:23.973506 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:23.973443 4971 scope.go:122] "RemoveContainer" containerID="cf0e905caf6b8f080ea89a3b15179ee7a8fd6725248ac002b2e7046fa6458ae1" Nov 02 20:30:23.973706 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:23.973686 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd_odh-ai-gateway-infra(fd4e27eb-15d7-48c0-8612-c605fac7fe38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" podUID="fd4e27eb-15d7-48c0-8612-c605fac7fe38" Nov 02 20:30:23.975231 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:23.975212 4971 generic.go:356] "Generic (PLEG): container finished" podID="853568b4-1289-42dc-aaa8-1f0654101bea" containerID="4710e23580fb47baa9439421d8f7cf4dbb1eae9a79dee7532a53953fd08247b3" exitCode=60 Nov 02 20:30:23.975326 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:23.975282 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" event={"ID":"853568b4-1289-42dc-aaa8-1f0654101bea","Type":"ContainerDied","Data":"4710e23580fb47baa9439421d8f7cf4dbb1eae9a79dee7532a53953fd08247b3"} Nov 02 20:30:23.975602 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:23.975582 4971 scope.go:122] "RemoveContainer" containerID="4710e23580fb47baa9439421d8f7cf4dbb1eae9a79dee7532a53953fd08247b3" Nov 02 20:30:23.975882 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:23.975861 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk_odh-ai-gateway-infra(853568b4-1289-42dc-aaa8-1f0654101bea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" podUID="853568b4-1289-42dc-aaa8-1f0654101bea" Nov 02 20:30:23.987774 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:23.987748 4971 scope.go:122] "RemoveContainer" containerID="36fae56df1a388ddd588e76b30b91d7ab491b8f38e2fa467d0a8a5f994768491" Nov 02 20:30:24.099262 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.099224 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd"] Nov 02 20:30:24.187509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.187476 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk"] Nov 02 20:30:24.297134 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.297100 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp"] Nov 02 20:30:24.455558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.455535 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" Nov 02 20:30:24.533276 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.533252 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52ebc70-c0d2-440a-951b-741be403aced-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52ebc70-c0d2-440a-951b-741be403aced-maas-api-service-ca\") pod \"c52ebc70-c0d2-440a-951b-741be403aced\" (UID: \"c52ebc70-c0d2-440a-951b-741be403aced\") " Nov 02 20:30:24.533423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.533288 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52ebc70-c0d2-440a-951b-741be403aced-kube-api-access-2mwxl\" (UniqueName: \"kubernetes.io/projected/c52ebc70-c0d2-440a-951b-741be403aced-kube-api-access-2mwxl\") pod \"c52ebc70-c0d2-440a-951b-741be403aced\" (UID: \"c52ebc70-c0d2-440a-951b-741be403aced\") " Nov 02 20:30:24.533576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.533555 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52ebc70-c0d2-440a-951b-741be403aced-maas-api-service-ca" pod "c52ebc70-c0d2-440a-951b-741be403aced" (UID: "c52ebc70-c0d2-440a-951b-741be403aced"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:24.535673 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.535650 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52ebc70-c0d2-440a-951b-741be403aced-kube-api-access-2mwxl" pod "c52ebc70-c0d2-440a-951b-741be403aced" (UID: "c52ebc70-c0d2-440a-951b-741be403aced"). InnerVolumeSpecName "kube-api-access-2mwxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:24.634672 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.634641 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52ebc70-c0d2-440a-951b-741be403aced-maas-api-service-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:24.634770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.634674 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mwxl\" (UniqueName: \"kubernetes.io/projected/c52ebc70-c0d2-440a-951b-741be403aced-kube-api-access-2mwxl\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:24.859526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.859502 4971 scope.go:122] "RemoveContainer" containerID="dde75b1d594e6a959f5979c6b4155a948ba6e0eaae82e8989e9dc6447bf5d6c7" Nov 02 20:30:24.987828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.987797 4971 generic.go:356] "Generic (PLEG): container finished" podID="8a48b92a-6991-485e-8d41-de59766de14e" containerID="ab1d78ceb514e83cc1bf92248bfcc6108b5ed93f9acd54d359b86d11094f56c0" exitCode=60 Nov 02 20:30:24.988292 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.987837 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" event={"ID":"8a48b92a-6991-485e-8d41-de59766de14e","Type":"ContainerDied","Data":"ab1d78ceb514e83cc1bf92248bfcc6108b5ed93f9acd54d359b86d11094f56c0"} Nov 02 20:30:24.988292 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.987917 4971 scope.go:122] "RemoveContainer" containerID="310ea0b11e3c1c95d6cfd7714f50367f7f0503ac0d90aab6657e3da01ca442f8" Nov 02 20:30:24.988478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.988301 4971 scope.go:122] "RemoveContainer" containerID="ab1d78ceb514e83cc1bf92248bfcc6108b5ed93f9acd54d359b86d11094f56c0" Nov 02 20:30:24.988578 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:24.988554 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x_odh-ai-gateway-infra(8a48b92a-6991-485e-8d41-de59766de14e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" podUID="8a48b92a-6991-485e-8d41-de59766de14e" Nov 02 20:30:24.994300 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.994273 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" event={"ID":"c52ebc70-c0d2-440a-951b-741be403aced","Type":"ContainerDied","Data":"de112b124261846c3d4a1cc92cf902ce94beb513c2ecba0aace7bb6c837eebd1"} Nov 02 20:30:24.994456 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:24.994323 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp" Nov 02 20:30:25.008379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.008361 4971 scope.go:122] "RemoveContainer" containerID="7562dc4183356849ebc2a856b24e89eb5d0091a7bf77a8d8ff8549f648d5fafa" Nov 02 20:30:25.142405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.142370 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" Nov 02 20:30:25.176752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.176729 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" Nov 02 20:30:25.242363 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.242340 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd4e27eb-15d7-48c0-8612-c605fac7fe38-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd4e27eb-15d7-48c0-8612-c605fac7fe38-maas-api-service-ca\") pod \"fd4e27eb-15d7-48c0-8612-c605fac7fe38\" (UID: \"fd4e27eb-15d7-48c0-8612-c605fac7fe38\") " Nov 02 20:30:25.242459 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.242407 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/853568b4-1289-42dc-aaa8-1f0654101bea-kube-api-access-z58zx\" (UniqueName: \"kubernetes.io/projected/853568b4-1289-42dc-aaa8-1f0654101bea-kube-api-access-z58zx\") pod \"853568b4-1289-42dc-aaa8-1f0654101bea\" (UID: \"853568b4-1289-42dc-aaa8-1f0654101bea\") " Nov 02 20:30:25.242503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.242494 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd4e27eb-15d7-48c0-8612-c605fac7fe38-kube-api-access-qh46x\" (UniqueName: \"kubernetes.io/projected/fd4e27eb-15d7-48c0-8612-c605fac7fe38-kube-api-access-qh46x\") pod \"fd4e27eb-15d7-48c0-8612-c605fac7fe38\" (UID: \"fd4e27eb-15d7-48c0-8612-c605fac7fe38\") " Nov 02 20:30:25.242559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.242547 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/853568b4-1289-42dc-aaa8-1f0654101bea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/853568b4-1289-42dc-aaa8-1f0654101bea-maas-api-service-ca\") pod \"853568b4-1289-42dc-aaa8-1f0654101bea\" (UID: \"853568b4-1289-42dc-aaa8-1f0654101bea\") " Nov 02 20:30:25.242710 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.242683 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd4e27eb-15d7-48c0-8612-c605fac7fe38-maas-api-service-ca" pod "fd4e27eb-15d7-48c0-8612-c605fac7fe38" (UID: "fd4e27eb-15d7-48c0-8612-c605fac7fe38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:25.242916 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.242897 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/853568b4-1289-42dc-aaa8-1f0654101bea-maas-api-service-ca" pod "853568b4-1289-42dc-aaa8-1f0654101bea" (UID: "853568b4-1289-42dc-aaa8-1f0654101bea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:25.242991 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.242959 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd4e27eb-15d7-48c0-8612-c605fac7fe38-maas-api-service-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:25.244464 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.244445 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/853568b4-1289-42dc-aaa8-1f0654101bea-kube-api-access-z58zx" pod "853568b4-1289-42dc-aaa8-1f0654101bea" (UID: "853568b4-1289-42dc-aaa8-1f0654101bea"). InnerVolumeSpecName "kube-api-access-z58zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:25.244733 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.244714 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd4e27eb-15d7-48c0-8612-c605fac7fe38-kube-api-access-qh46x" pod "fd4e27eb-15d7-48c0-8612-c605fac7fe38" (UID: "fd4e27eb-15d7-48c0-8612-c605fac7fe38"). InnerVolumeSpecName "kube-api-access-qh46x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:25.344235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.344211 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh46x\" (UniqueName: \"kubernetes.io/projected/fd4e27eb-15d7-48c0-8612-c605fac7fe38-kube-api-access-qh46x\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:25.344235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.344234 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/853568b4-1289-42dc-aaa8-1f0654101bea-maas-api-service-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:25.344393 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.344247 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z58zx\" (UniqueName: \"kubernetes.io/projected/853568b4-1289-42dc-aaa8-1f0654101bea-kube-api-access-z58zx\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:25.418886 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.418819 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp"] Nov 02 20:30:25.423488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.423463 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-45dkp"] Nov 02 20:30:25.859518 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.859495 4971 scope.go:122] "RemoveContainer" containerID="45fe220b550c195dc865d94ed23269a266d477cfc462e94eb2e7c4b545cb4214" Nov 02 20:30:25.867671 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:25.867646 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c52ebc70-c0d2-440a-951b-741be403aced" path="/var/lib/kubelet/pods/c52ebc70-c0d2-440a-951b-741be403aced/volumes" Nov 02 20:30:26.001791 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.001719 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" event={"ID":"fd4e27eb-15d7-48c0-8612-c605fac7fe38","Type":"ContainerDied","Data":"3894270354d136501b5b8a0c331ea38924fcd4d0d2501d313f34267f895c9660"} Nov 02 20:30:26.001791 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.001789 4971 scope.go:122] "RemoveContainer" containerID="cf0e905caf6b8f080ea89a3b15179ee7a8fd6725248ac002b2e7046fa6458ae1" Nov 02 20:30:26.002349 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.001892 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd" Nov 02 20:30:26.006444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.006425 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" Nov 02 20:30:26.006823 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.006784 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk" event={"ID":"853568b4-1289-42dc-aaa8-1f0654101bea","Type":"ContainerDied","Data":"18159782e9093b3d611fe5fa418fb0deddf419dbb92928cd20be443d7c2ed333"} Nov 02 20:30:26.019064 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.017655 4971 generic.go:356] "Generic (PLEG): container finished" podID="0f2e0cfc-b518-4daa-933f-10cb82d72184" containerID="6127e286ba9ee39c2475d1cae6ed84ce7931181165be44b6f4ffd7449964f0b7" exitCode=60 Nov 02 20:30:26.019064 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.017942 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" event={"ID":"0f2e0cfc-b518-4daa-933f-10cb82d72184","Type":"ContainerDied","Data":"6127e286ba9ee39c2475d1cae6ed84ce7931181165be44b6f4ffd7449964f0b7"} Nov 02 20:30:26.019064 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.018517 4971 scope.go:122] "RemoveContainer" containerID="6127e286ba9ee39c2475d1cae6ed84ce7931181165be44b6f4ffd7449964f0b7" Nov 02 20:30:26.019064 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:26.018864 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f_odh-ai-gateway-infra(0f2e0cfc-b518-4daa-933f-10cb82d72184)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" podUID="0f2e0cfc-b518-4daa-933f-10cb82d72184" Nov 02 20:30:26.021065 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.020968 4971 scope.go:122] "RemoveContainer" containerID="4710e23580fb47baa9439421d8f7cf4dbb1eae9a79dee7532a53953fd08247b3" Nov 02 20:30:26.042192 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.042174 4971 scope.go:122] "RemoveContainer" containerID="dde75b1d594e6a959f5979c6b4155a948ba6e0eaae82e8989e9dc6447bf5d6c7" Nov 02 20:30:26.107862 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.107836 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x"] Nov 02 20:30:26.157975 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.157947 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk"] Nov 02 20:30:26.164429 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.164405 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a5cad4-29894190-wspdk"] Nov 02 20:30:26.267656 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.267632 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd"] Nov 02 20:30:26.273387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.273368 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" Nov 02 20:30:26.273498 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.273393 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894190-s2rbd"] Nov 02 20:30:26.354868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.354842 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a48b92a-6991-485e-8d41-de59766de14e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a48b92a-6991-485e-8d41-de59766de14e-maas-api-service-ca\") pod \"8a48b92a-6991-485e-8d41-de59766de14e\" (UID: \"8a48b92a-6991-485e-8d41-de59766de14e\") " Nov 02 20:30:26.355045 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.354983 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a48b92a-6991-485e-8d41-de59766de14e-kube-api-access-c68zm\" (UniqueName: \"kubernetes.io/projected/8a48b92a-6991-485e-8d41-de59766de14e-kube-api-access-c68zm\") pod \"8a48b92a-6991-485e-8d41-de59766de14e\" (UID: \"8a48b92a-6991-485e-8d41-de59766de14e\") " Nov 02 20:30:26.355214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.355189 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a48b92a-6991-485e-8d41-de59766de14e-maas-api-service-ca" pod "8a48b92a-6991-485e-8d41-de59766de14e" (UID: "8a48b92a-6991-485e-8d41-de59766de14e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:26.355440 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.355422 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a48b92a-6991-485e-8d41-de59766de14e-maas-api-service-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:26.357248 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.357228 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a48b92a-6991-485e-8d41-de59766de14e-kube-api-access-c68zm" pod "8a48b92a-6991-485e-8d41-de59766de14e" (UID: "8a48b92a-6991-485e-8d41-de59766de14e"). InnerVolumeSpecName "kube-api-access-c68zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:26.456719 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.456699 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c68zm\" (UniqueName: \"kubernetes.io/projected/8a48b92a-6991-485e-8d41-de59766de14e-kube-api-access-c68zm\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:26.923438 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:26.923404 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:27.031723 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.031653 4971 generic.go:356] "Generic (PLEG): container finished" podID="c239e090-9178-4ab1-a71d-dd83391df358" containerID="e9592c37ba6cb0ae281aa60481a655f5caa2f572f7833e2f648e05a1de495bd3" exitCode=60 Nov 02 20:30:27.032169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.031728 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" event={"ID":"c239e090-9178-4ab1-a71d-dd83391df358","Type":"ContainerDied","Data":"e9592c37ba6cb0ae281aa60481a655f5caa2f572f7833e2f648e05a1de495bd3"} Nov 02 20:30:27.032169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.031797 4971 scope.go:122] "RemoveContainer" containerID="45fe220b550c195dc865d94ed23269a266d477cfc462e94eb2e7c4b545cb4214" Nov 02 20:30:27.032359 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.032342 4971 scope.go:122] "RemoveContainer" containerID="e9592c37ba6cb0ae281aa60481a655f5caa2f572f7833e2f648e05a1de495bd3" Nov 02 20:30:27.032671 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:27.032647 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s_odh-ai-gateway-infra(c239e090-9178-4ab1-a71d-dd83391df358)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" podUID="c239e090-9178-4ab1-a71d-dd83391df358" Nov 02 20:30:27.033481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.033462 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" Nov 02 20:30:27.033564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.033485 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x" event={"ID":"8a48b92a-6991-485e-8d41-de59766de14e","Type":"ContainerDied","Data":"2153ddd388f1a1908219275b43cef4e571eb13b100ec30f313cc7d5a14e21fce"} Nov 02 20:30:27.044790 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.044774 4971 scope.go:122] "RemoveContainer" containerID="ab1d78ceb514e83cc1bf92248bfcc6108b5ed93f9acd54d359b86d11094f56c0" Nov 02 20:30:27.172675 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.172623 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:27.247347 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.247320 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x"] Nov 02 20:30:27.251244 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.251218 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894190-mz44x"] Nov 02 20:30:27.389752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.389721 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f"] Nov 02 20:30:27.457757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.457528 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:27.471264 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.471236 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:27.560551 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.560529 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" Nov 02 20:30:27.676222 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.676162 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2e0cfc-b518-4daa-933f-10cb82d72184-kube-api-access-qvrjj\" (UniqueName: \"kubernetes.io/projected/0f2e0cfc-b518-4daa-933f-10cb82d72184-kube-api-access-qvrjj\") pod \"0f2e0cfc-b518-4daa-933f-10cb82d72184\" (UID: \"0f2e0cfc-b518-4daa-933f-10cb82d72184\") " Nov 02 20:30:27.676222 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.676216 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f2e0cfc-b518-4daa-933f-10cb82d72184-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2e0cfc-b518-4daa-933f-10cb82d72184-maas-api-service-ca\") pod \"0f2e0cfc-b518-4daa-933f-10cb82d72184\" (UID: \"0f2e0cfc-b518-4daa-933f-10cb82d72184\") " Nov 02 20:30:27.676568 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.676545 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2e0cfc-b518-4daa-933f-10cb82d72184-maas-api-service-ca" pod "0f2e0cfc-b518-4daa-933f-10cb82d72184" (UID: "0f2e0cfc-b518-4daa-933f-10cb82d72184"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:27.678444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.678424 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2e0cfc-b518-4daa-933f-10cb82d72184-kube-api-access-qvrjj" pod "0f2e0cfc-b518-4daa-933f-10cb82d72184" (UID: "0f2e0cfc-b518-4daa-933f-10cb82d72184"). InnerVolumeSpecName "kube-api-access-qvrjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:27.777581 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.777553 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvrjj\" (UniqueName: \"kubernetes.io/projected/0f2e0cfc-b518-4daa-933f-10cb82d72184-kube-api-access-qvrjj\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:27.777581 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.777580 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f2e0cfc-b518-4daa-933f-10cb82d72184-maas-api-service-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:27.866539 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.866511 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="853568b4-1289-42dc-aaa8-1f0654101bea" path="/var/lib/kubelet/pods/853568b4-1289-42dc-aaa8-1f0654101bea/volumes" Nov 02 20:30:27.866891 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.866878 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a48b92a-6991-485e-8d41-de59766de14e" path="/var/lib/kubelet/pods/8a48b92a-6991-485e-8d41-de59766de14e/volumes" Nov 02 20:30:27.867266 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:27.867252 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd4e27eb-15d7-48c0-8612-c605fac7fe38" path="/var/lib/kubelet/pods/fd4e27eb-15d7-48c0-8612-c605fac7fe38/volumes" Nov 02 20:30:28.044111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:28.044038 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" Nov 02 20:30:28.044111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:28.044073 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f" event={"ID":"0f2e0cfc-b518-4daa-933f-10cb82d72184","Type":"ContainerDied","Data":"9d2d918626732d8f919eb678570aeb4a36ddcd1f3071cb13b2f43a41dc118e20"} Nov 02 20:30:28.044547 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:28.044136 4971 scope.go:122] "RemoveContainer" containerID="6127e286ba9ee39c2475d1cae6ed84ce7931181165be44b6f4ffd7449964f0b7" Nov 02 20:30:28.059282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:28.059258 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:30:28.151294 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:28.151265 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s"] Nov 02 20:30:28.199047 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:28.199022 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f"] Nov 02 20:30:28.203235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:28.203206 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894190-k7n9f"] Nov 02 20:30:28.313831 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:28.313771 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" Nov 02 20:30:28.385438 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:28.385408 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c239e090-9178-4ab1-a71d-dd83391df358-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c239e090-9178-4ab1-a71d-dd83391df358-maas-api-service-ca\") pod \"c239e090-9178-4ab1-a71d-dd83391df358\" (UID: \"c239e090-9178-4ab1-a71d-dd83391df358\") " Nov 02 20:30:28.385580 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:28.385507 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c239e090-9178-4ab1-a71d-dd83391df358-kube-api-access-7xbx9\" (UniqueName: \"kubernetes.io/projected/c239e090-9178-4ab1-a71d-dd83391df358-kube-api-access-7xbx9\") pod \"c239e090-9178-4ab1-a71d-dd83391df358\" (UID: \"c239e090-9178-4ab1-a71d-dd83391df358\") " Nov 02 20:30:28.385812 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:28.385790 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c239e090-9178-4ab1-a71d-dd83391df358-maas-api-service-ca" pod "c239e090-9178-4ab1-a71d-dd83391df358" (UID: "c239e090-9178-4ab1-a71d-dd83391df358"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:28.385936 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:28.385921 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c239e090-9178-4ab1-a71d-dd83391df358-maas-api-service-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:28.387626 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:28.387598 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c239e090-9178-4ab1-a71d-dd83391df358-kube-api-access-7xbx9" pod "c239e090-9178-4ab1-a71d-dd83391df358" (UID: "c239e090-9178-4ab1-a71d-dd83391df358"). InnerVolumeSpecName "kube-api-access-7xbx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:28.487028 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:28.486996 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xbx9\" (UniqueName: \"kubernetes.io/projected/c239e090-9178-4ab1-a71d-dd83391df358-kube-api-access-7xbx9\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:29.056347 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:29.056321 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" Nov 02 20:30:29.056807 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:29.056359 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s" event={"ID":"c239e090-9178-4ab1-a71d-dd83391df358","Type":"ContainerDied","Data":"be86298b9014a8053b2a8f4e617f395aaa9b25abbd13446592cf79d8a7a9277e"} Nov 02 20:30:29.056807 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:29.056411 4971 scope.go:122] "RemoveContainer" containerID="e9592c37ba6cb0ae281aa60481a655f5caa2f572f7833e2f648e05a1de495bd3" Nov 02 20:30:29.215617 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:29.215542 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s"] Nov 02 20:30:29.225536 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:29.225501 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30ee89-29894190-8bn8s"] Nov 02 20:30:29.873644 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:29.873604 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2e0cfc-b518-4daa-933f-10cb82d72184" path="/var/lib/kubelet/pods/0f2e0cfc-b518-4daa-933f-10cb82d72184/volumes" Nov 02 20:30:29.874163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:29.874144 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c239e090-9178-4ab1-a71d-dd83391df358" path="/var/lib/kubelet/pods/c239e090-9178-4ab1-a71d-dd83391df358/volumes" Nov 02 20:30:31.504580 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:31.504549 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:31.748952 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:31.748917 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:33.338569 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:30:33.338529 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:30:36.497198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:36.497163 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:41.292514 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:41.292468 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:42.713154 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:42.713124 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:46.133154 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:46.133109 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:50.929163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:50.929122 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:55.704977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:55.704924 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:30:57.971857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.971823 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k"] Nov 02 20:30:57.972485 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972463 4971 cpu_manager.go:414] "containerMap: removing container" podUID="853568b4-1289-42dc-aaa8-1f0654101bea" containerName="cleanup" Nov 02 20:30:57.972573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972487 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="853568b4-1289-42dc-aaa8-1f0654101bea" containerName="cleanup" Nov 02 20:30:57.972573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972523 4971 cpu_manager.go:414] "containerMap: removing container" podUID="8a48b92a-6991-485e-8d41-de59766de14e" containerName="cleanup" Nov 02 20:30:57.972573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972531 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a48b92a-6991-485e-8d41-de59766de14e" containerName="cleanup" Nov 02 20:30:57.972573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972547 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c239e090-9178-4ab1-a71d-dd83391df358" containerName="cleanup" Nov 02 20:30:57.972573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972554 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c239e090-9178-4ab1-a71d-dd83391df358" containerName="cleanup" Nov 02 20:30:57.972573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972564 4971 cpu_manager.go:414] "containerMap: removing container" podUID="0f2e0cfc-b518-4daa-933f-10cb82d72184" containerName="cleanup" Nov 02 20:30:57.972573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972572 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2e0cfc-b518-4daa-933f-10cb82d72184" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972583 4971 cpu_manager.go:414] "containerMap: removing container" podUID="fd4e27eb-15d7-48c0-8612-c605fac7fe38" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972589 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4e27eb-15d7-48c0-8612-c605fac7fe38" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972600 4971 cpu_manager.go:414] "containerMap: removing container" podUID="fd4e27eb-15d7-48c0-8612-c605fac7fe38" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972607 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4e27eb-15d7-48c0-8612-c605fac7fe38" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972618 4971 cpu_manager.go:414] "containerMap: removing container" podUID="8a48b92a-6991-485e-8d41-de59766de14e" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972624 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a48b92a-6991-485e-8d41-de59766de14e" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972640 4971 cpu_manager.go:414] "containerMap: removing container" podUID="0f2e0cfc-b518-4daa-933f-10cb82d72184" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972646 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2e0cfc-b518-4daa-933f-10cb82d72184" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972658 4971 cpu_manager.go:414] "containerMap: removing container" podUID="0f2e0cfc-b518-4daa-933f-10cb82d72184" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972664 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2e0cfc-b518-4daa-933f-10cb82d72184" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972679 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c52ebc70-c0d2-440a-951b-741be403aced" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972686 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52ebc70-c0d2-440a-951b-741be403aced" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972696 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c239e090-9178-4ab1-a71d-dd83391df358" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972701 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c239e090-9178-4ab1-a71d-dd83391df358" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972709 4971 cpu_manager.go:414] "containerMap: removing container" podUID="853568b4-1289-42dc-aaa8-1f0654101bea" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972715 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="853568b4-1289-42dc-aaa8-1f0654101bea" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972734 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c52ebc70-c0d2-440a-951b-741be403aced" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972740 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52ebc70-c0d2-440a-951b-741be403aced" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972758 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c52ebc70-c0d2-440a-951b-741be403aced" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972765 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52ebc70-c0d2-440a-951b-741be403aced" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972774 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c239e090-9178-4ab1-a71d-dd83391df358" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972780 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c239e090-9178-4ab1-a71d-dd83391df358" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972791 4971 cpu_manager.go:414] "containerMap: removing container" podUID="8a48b92a-6991-485e-8d41-de59766de14e" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972798 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a48b92a-6991-485e-8d41-de59766de14e" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972926 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd4e27eb-15d7-48c0-8612-c605fac7fe38" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972936 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a48b92a-6991-485e-8d41-de59766de14e" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972946 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2e0cfc-b518-4daa-933f-10cb82d72184" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972954 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2e0cfc-b518-4daa-933f-10cb82d72184" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972966 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd4e27eb-15d7-48c0-8612-c605fac7fe38" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972973 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52ebc70-c0d2-440a-951b-741be403aced" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972982 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c239e090-9178-4ab1-a71d-dd83391df358" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972989 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="853568b4-1289-42dc-aaa8-1f0654101bea" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.972999 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52ebc70-c0d2-440a-951b-741be403aced" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.973009 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52ebc70-c0d2-440a-951b-741be403aced" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.973019 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c239e090-9178-4ab1-a71d-dd83391df358" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.973027 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a48b92a-6991-485e-8d41-de59766de14e" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.973039 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a48b92a-6991-485e-8d41-de59766de14e" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.973048 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c239e090-9178-4ab1-a71d-dd83391df358" containerName="cleanup" Nov 02 20:30:57.973032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.973057 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="853568b4-1289-42dc-aaa8-1f0654101bea" containerName="cleanup" Nov 02 20:30:57.977182 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:57.977154 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" Nov 02 20:30:58.028392 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:58.026846 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:30:58.030262 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:58.030240 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k"] Nov 02 20:30:58.073198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:58.073166 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:30:58.121111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:58.116330 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2de4b0-146a-4f77-ac02-43b51df1dea9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bbb4597-fgq4k\" (UID: \"dc2de4b0-146a-4f77-ac02-43b51df1dea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" Nov 02 20:30:58.121111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:58.116503 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx7dj\" (UniqueName: \"kubernetes.io/projected/dc2de4b0-146a-4f77-ac02-43b51df1dea9-kube-api-access-vx7dj\") pod \"maas-api-revoke-sub-0bbb4597-fgq4k\" (UID: \"dc2de4b0-146a-4f77-ac02-43b51df1dea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" Nov 02 20:30:58.121111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:58.116539 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2de4b0-146a-4f77-ac02-43b51df1dea9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bbb4597-fgq4k\" (UID: \"dc2de4b0-146a-4f77-ac02-43b51df1dea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" Nov 02 20:30:58.217394 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:58.217360 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2de4b0-146a-4f77-ac02-43b51df1dea9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bbb4597-fgq4k\" (UID: \"dc2de4b0-146a-4f77-ac02-43b51df1dea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" Nov 02 20:30:58.217589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:58.217529 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx7dj\" (UniqueName: \"kubernetes.io/projected/dc2de4b0-146a-4f77-ac02-43b51df1dea9-kube-api-access-vx7dj\") pod \"maas-api-revoke-sub-0bbb4597-fgq4k\" (UID: \"dc2de4b0-146a-4f77-ac02-43b51df1dea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" Nov 02 20:30:58.217589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:58.217568 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2de4b0-146a-4f77-ac02-43b51df1dea9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bbb4597-fgq4k\" (UID: \"dc2de4b0-146a-4f77-ac02-43b51df1dea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" Nov 02 20:30:58.218284 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:58.218260 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2de4b0-146a-4f77-ac02-43b51df1dea9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bbb4597-fgq4k\" (UID: \"dc2de4b0-146a-4f77-ac02-43b51df1dea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" Nov 02 20:30:58.219954 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:58.219929 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2de4b0-146a-4f77-ac02-43b51df1dea9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bbb4597-fgq4k\" (UID: \"dc2de4b0-146a-4f77-ac02-43b51df1dea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" Nov 02 20:30:58.275892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:58.275822 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx7dj\" (UniqueName: \"kubernetes.io/projected/dc2de4b0-146a-4f77-ac02-43b51df1dea9-kube-api-access-vx7dj\") pod \"maas-api-revoke-sub-0bbb4597-fgq4k\" (UID: \"dc2de4b0-146a-4f77-ac02-43b51df1dea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" Nov 02 20:30:58.292243 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:58.292212 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" Nov 02 20:30:58.707320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:58.706697 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k"] Nov 02 20:30:59.266852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:59.266746 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" event={"ID":"dc2de4b0-146a-4f77-ac02-43b51df1dea9","Type":"ContainerStarted","Data":"36fbd1ce92fcd73a96145a1df36c32a0e3c77d90715248bab9d0479a1eae8256"} Nov 02 20:30:59.266852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:59.266827 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" event={"ID":"dc2de4b0-146a-4f77-ac02-43b51df1dea9","Type":"ContainerStarted","Data":"21214b23d0c3c9554a66e58c5b8690885e775d402ec573fef560e81c9532e25a"} Nov 02 20:30:59.376710 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:30:59.376660 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" podStartSLOduration=2.376643542 podStartE2EDuration="2.376643542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:59.375398559 +0000 UTC m=+987.665055087" watchObservedRunningTime="2026-11-02 20:30:59.376643542 +0000 UTC m=+987.666300070" Nov 02 20:31:00.275413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:00.275375 4971 generic.go:356] "Generic (PLEG): container finished" podID="dc2de4b0-146a-4f77-ac02-43b51df1dea9" containerID="36fbd1ce92fcd73a96145a1df36c32a0e3c77d90715248bab9d0479a1eae8256" exitCode=0 Nov 02 20:31:00.275935 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:00.275415 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" event={"ID":"dc2de4b0-146a-4f77-ac02-43b51df1dea9","Type":"ContainerDied","Data":"36fbd1ce92fcd73a96145a1df36c32a0e3c77d90715248bab9d0479a1eae8256"} Nov 02 20:31:00.528831 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:00.528751 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:01.425021 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:01.424987 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" Nov 02 20:31:01.571801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:01.571713 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc2de4b0-146a-4f77-ac02-43b51df1dea9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2de4b0-146a-4f77-ac02-43b51df1dea9-maas-api-openshift-ca\") pod \"dc2de4b0-146a-4f77-ac02-43b51df1dea9\" (UID: \"dc2de4b0-146a-4f77-ac02-43b51df1dea9\") " Nov 02 20:31:01.571801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:01.571761 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc2de4b0-146a-4f77-ac02-43b51df1dea9-kube-api-access-vx7dj\" (UniqueName: \"kubernetes.io/projected/dc2de4b0-146a-4f77-ac02-43b51df1dea9-kube-api-access-vx7dj\") pod \"dc2de4b0-146a-4f77-ac02-43b51df1dea9\" (UID: \"dc2de4b0-146a-4f77-ac02-43b51df1dea9\") " Nov 02 20:31:01.572032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:01.571922 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc2de4b0-146a-4f77-ac02-43b51df1dea9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2de4b0-146a-4f77-ac02-43b51df1dea9-maas-api-xks-ca\") pod \"dc2de4b0-146a-4f77-ac02-43b51df1dea9\" (UID: \"dc2de4b0-146a-4f77-ac02-43b51df1dea9\") " Nov 02 20:31:01.572174 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:01.572145 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc2de4b0-146a-4f77-ac02-43b51df1dea9-maas-api-openshift-ca" pod "dc2de4b0-146a-4f77-ac02-43b51df1dea9" (UID: "dc2de4b0-146a-4f77-ac02-43b51df1dea9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:01.572400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:01.572376 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2de4b0-146a-4f77-ac02-43b51df1dea9-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:01.574295 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:01.574271 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc2de4b0-146a-4f77-ac02-43b51df1dea9-maas-api-xks-ca" pod "dc2de4b0-146a-4f77-ac02-43b51df1dea9" (UID: "dc2de4b0-146a-4f77-ac02-43b51df1dea9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:01.574578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:01.574552 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2de4b0-146a-4f77-ac02-43b51df1dea9-kube-api-access-vx7dj" pod "dc2de4b0-146a-4f77-ac02-43b51df1dea9" (UID: "dc2de4b0-146a-4f77-ac02-43b51df1dea9"). InnerVolumeSpecName "kube-api-access-vx7dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:01.672982 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:01.672955 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx7dj\" (UniqueName: \"kubernetes.io/projected/dc2de4b0-146a-4f77-ac02-43b51df1dea9-kube-api-access-vx7dj\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:01.672982 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:01.672979 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2de4b0-146a-4f77-ac02-43b51df1dea9-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:02.288714 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:02.288633 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" Nov 02 20:31:02.288714 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:02.288658 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k" event={"ID":"dc2de4b0-146a-4f77-ac02-43b51df1dea9","Type":"ContainerDied","Data":"21214b23d0c3c9554a66e58c5b8690885e775d402ec573fef560e81c9532e25a"} Nov 02 20:31:02.288900 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:02.288726 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21214b23d0c3c9554a66e58c5b8690885e775d402ec573fef560e81c9532e25a" Nov 02 20:31:03.336976 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:31:03.336944 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:31:03.687296 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:03.687266 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:03.938999 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:03.938896 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:05.505374 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:05.505337 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:10.346495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:10.346461 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:13.373362 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.373332 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm"] Nov 02 20:31:13.373848 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.373834 4971 cpu_manager.go:414] "containerMap: removing container" podUID="dc2de4b0-146a-4f77-ac02-43b51df1dea9" containerName="revoke-keys" Nov 02 20:31:13.373892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.373848 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc2de4b0-146a-4f77-ac02-43b51df1dea9" containerName="revoke-keys" Nov 02 20:31:13.373892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.373861 4971 cpu_manager.go:414] "containerMap: removing container" podUID="fd4e27eb-15d7-48c0-8612-c605fac7fe38" containerName="cleanup" Nov 02 20:31:13.373892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.373866 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4e27eb-15d7-48c0-8612-c605fac7fe38" containerName="cleanup" Nov 02 20:31:13.373997 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.373896 4971 cpu_manager.go:414] "containerMap: removing container" podUID="853568b4-1289-42dc-aaa8-1f0654101bea" containerName="cleanup" Nov 02 20:31:13.373997 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.373901 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="853568b4-1289-42dc-aaa8-1f0654101bea" containerName="cleanup" Nov 02 20:31:13.373997 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.373995 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2e0cfc-b518-4daa-933f-10cb82d72184" containerName="cleanup" Nov 02 20:31:13.374140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.374010 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc2de4b0-146a-4f77-ac02-43b51df1dea9" containerName="revoke-keys" Nov 02 20:31:13.374140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.374016 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="853568b4-1289-42dc-aaa8-1f0654101bea" containerName="cleanup" Nov 02 20:31:13.374140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.374023 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd4e27eb-15d7-48c0-8612-c605fac7fe38" containerName="cleanup" Nov 02 20:31:13.377517 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.377502 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" Nov 02 20:31:13.474251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.474219 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:31:13.474251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.474225 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:31:13.478846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.478824 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm"] Nov 02 20:31:13.490948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.490922 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db07e7a0-a6ad-4613-9afe-656747fe386d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9d7e5c-ttpbm\" (UID: \"db07e7a0-a6ad-4613-9afe-656747fe386d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" Nov 02 20:31:13.491107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.490965 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zcld\" (UniqueName: \"kubernetes.io/projected/db07e7a0-a6ad-4613-9afe-656747fe386d-kube-api-access-7zcld\") pod \"maas-api-revoke-sub-1c9d7e5c-ttpbm\" (UID: \"db07e7a0-a6ad-4613-9afe-656747fe386d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" Nov 02 20:31:13.491187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.491126 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db07e7a0-a6ad-4613-9afe-656747fe386d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9d7e5c-ttpbm\" (UID: \"db07e7a0-a6ad-4613-9afe-656747fe386d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" Nov 02 20:31:13.600602 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.592858 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db07e7a0-a6ad-4613-9afe-656747fe386d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9d7e5c-ttpbm\" (UID: \"db07e7a0-a6ad-4613-9afe-656747fe386d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" Nov 02 20:31:13.600602 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.592933 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zcld\" (UniqueName: \"kubernetes.io/projected/db07e7a0-a6ad-4613-9afe-656747fe386d-kube-api-access-7zcld\") pod \"maas-api-revoke-sub-1c9d7e5c-ttpbm\" (UID: \"db07e7a0-a6ad-4613-9afe-656747fe386d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" Nov 02 20:31:13.600602 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.593024 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db07e7a0-a6ad-4613-9afe-656747fe386d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9d7e5c-ttpbm\" (UID: \"db07e7a0-a6ad-4613-9afe-656747fe386d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" Nov 02 20:31:13.600602 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.596163 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db07e7a0-a6ad-4613-9afe-656747fe386d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9d7e5c-ttpbm\" (UID: \"db07e7a0-a6ad-4613-9afe-656747fe386d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" Nov 02 20:31:13.600602 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.596768 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db07e7a0-a6ad-4613-9afe-656747fe386d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9d7e5c-ttpbm\" (UID: \"db07e7a0-a6ad-4613-9afe-656747fe386d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" Nov 02 20:31:13.655474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.655411 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zcld\" (UniqueName: \"kubernetes.io/projected/db07e7a0-a6ad-4613-9afe-656747fe386d-kube-api-access-7zcld\") pod \"maas-api-revoke-sub-1c9d7e5c-ttpbm\" (UID: \"db07e7a0-a6ad-4613-9afe-656747fe386d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" Nov 02 20:31:13.691155 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:13.691121 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" Nov 02 20:31:14.062123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:14.061194 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm"] Nov 02 20:31:14.063167 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:31:14.062466 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb07e7a0_a6ad_4613_9afe_656747fe386d.slice/crio-93106de5764e1268eebc94f258cde454b4dde5df59f8a0714550e12c400d3b4f WatchSource:0}: Error finding container 93106de5764e1268eebc94f258cde454b4dde5df59f8a0714550e12c400d3b4f: Status 404 returned error can't find the container with id 93106de5764e1268eebc94f258cde454b4dde5df59f8a0714550e12c400d3b4f Nov 02 20:31:14.360414 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:14.360384 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" event={"ID":"db07e7a0-a6ad-4613-9afe-656747fe386d","Type":"ContainerStarted","Data":"b5eed181be6d5ed7661e4fbf5c46fdf93b1ad20b3232c0a0c6d2659c16e67bb1"} Nov 02 20:31:14.360515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:14.360435 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" event={"ID":"db07e7a0-a6ad-4613-9afe-656747fe386d","Type":"ContainerStarted","Data":"93106de5764e1268eebc94f258cde454b4dde5df59f8a0714550e12c400d3b4f"} Nov 02 20:31:14.470993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:14.470942 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" podStartSLOduration=1.470929153 podStartE2EDuration="1.470929153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:14.470035066 +0000 UTC m=+1002.759691595" watchObservedRunningTime="2026-11-02 20:31:14.470929153 +0000 UTC m=+1002.760585680" Nov 02 20:31:15.099631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:15.099600 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:15.368308 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:15.368277 4971 generic.go:356] "Generic (PLEG): container finished" podID="db07e7a0-a6ad-4613-9afe-656747fe386d" containerID="b5eed181be6d5ed7661e4fbf5c46fdf93b1ad20b3232c0a0c6d2659c16e67bb1" exitCode=0 Nov 02 20:31:15.368504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:15.368372 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" event={"ID":"db07e7a0-a6ad-4613-9afe-656747fe386d","Type":"ContainerDied","Data":"b5eed181be6d5ed7661e4fbf5c46fdf93b1ad20b3232c0a0c6d2659c16e67bb1"} Nov 02 20:31:16.540846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:16.540822 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" Nov 02 20:31:16.734632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:16.734563 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db07e7a0-a6ad-4613-9afe-656747fe386d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db07e7a0-a6ad-4613-9afe-656747fe386d-maas-api-xks-ca\") pod \"db07e7a0-a6ad-4613-9afe-656747fe386d\" (UID: \"db07e7a0-a6ad-4613-9afe-656747fe386d\") " Nov 02 20:31:16.734632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:16.734604 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db07e7a0-a6ad-4613-9afe-656747fe386d-kube-api-access-7zcld\" (UniqueName: \"kubernetes.io/projected/db07e7a0-a6ad-4613-9afe-656747fe386d-kube-api-access-7zcld\") pod \"db07e7a0-a6ad-4613-9afe-656747fe386d\" (UID: \"db07e7a0-a6ad-4613-9afe-656747fe386d\") " Nov 02 20:31:16.734872 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:16.734686 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db07e7a0-a6ad-4613-9afe-656747fe386d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db07e7a0-a6ad-4613-9afe-656747fe386d-maas-api-openshift-ca\") pod \"db07e7a0-a6ad-4613-9afe-656747fe386d\" (UID: \"db07e7a0-a6ad-4613-9afe-656747fe386d\") " Nov 02 20:31:16.735164 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:16.735071 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db07e7a0-a6ad-4613-9afe-656747fe386d-maas-api-openshift-ca" pod "db07e7a0-a6ad-4613-9afe-656747fe386d" (UID: "db07e7a0-a6ad-4613-9afe-656747fe386d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:16.736896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:16.736872 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db07e7a0-a6ad-4613-9afe-656747fe386d-maas-api-xks-ca" pod "db07e7a0-a6ad-4613-9afe-656747fe386d" (UID: "db07e7a0-a6ad-4613-9afe-656747fe386d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:16.736992 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:16.736938 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db07e7a0-a6ad-4613-9afe-656747fe386d-kube-api-access-7zcld" pod "db07e7a0-a6ad-4613-9afe-656747fe386d" (UID: "db07e7a0-a6ad-4613-9afe-656747fe386d"). InnerVolumeSpecName "kube-api-access-7zcld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:16.836037 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:16.836013 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db07e7a0-a6ad-4613-9afe-656747fe386d-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:16.836037 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:16.836037 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zcld\" (UniqueName: \"kubernetes.io/projected/db07e7a0-a6ad-4613-9afe-656747fe386d-kube-api-access-7zcld\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:16.836215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:16.836047 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db07e7a0-a6ad-4613-9afe-656747fe386d-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:17.381676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:17.381644 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" Nov 02 20:31:17.381676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:17.381651 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm" event={"ID":"db07e7a0-a6ad-4613-9afe-656747fe386d","Type":"ContainerDied","Data":"93106de5764e1268eebc94f258cde454b4dde5df59f8a0714550e12c400d3b4f"} Nov 02 20:31:17.381880 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:17.381722 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93106de5764e1268eebc94f258cde454b4dde5df59f8a0714550e12c400d3b4f" Nov 02 20:31:19.198725 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.198698 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr"] Nov 02 20:31:19.199287 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.199272 4971 cpu_manager.go:414] "containerMap: removing container" podUID="db07e7a0-a6ad-4613-9afe-656747fe386d" containerName="revoke-keys" Nov 02 20:31:19.199330 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.199288 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db07e7a0-a6ad-4613-9afe-656747fe386d" containerName="revoke-keys" Nov 02 20:31:19.199386 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.199377 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="db07e7a0-a6ad-4613-9afe-656747fe386d" containerName="revoke-keys" Nov 02 20:31:19.202895 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.202877 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr" Nov 02 20:31:19.252672 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.252635 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:31:19.268708 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.268674 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e2a472-a91f-402d-bf02-3459102e377f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a370c93c-52pkr\" (UID: \"c9e2a472-a91f-402d-bf02-3459102e377f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr" Nov 02 20:31:19.268856 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.268828 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e2a472-a91f-402d-bf02-3459102e377f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a370c93c-52pkr\" (UID: \"c9e2a472-a91f-402d-bf02-3459102e377f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr" Nov 02 20:31:19.268907 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.268870 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl7mf\" (UniqueName: \"kubernetes.io/projected/c9e2a472-a91f-402d-bf02-3459102e377f-kube-api-access-dl7mf\") pod \"maas-api-revoke-sub-a370c93c-52pkr\" (UID: \"c9e2a472-a91f-402d-bf02-3459102e377f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr" Nov 02 20:31:19.299409 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.299381 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:31:19.302225 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.302197 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr"] Nov 02 20:31:19.370310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.370282 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e2a472-a91f-402d-bf02-3459102e377f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a370c93c-52pkr\" (UID: \"c9e2a472-a91f-402d-bf02-3459102e377f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr" Nov 02 20:31:19.370447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.370343 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl7mf\" (UniqueName: \"kubernetes.io/projected/c9e2a472-a91f-402d-bf02-3459102e377f-kube-api-access-dl7mf\") pod \"maas-api-revoke-sub-a370c93c-52pkr\" (UID: \"c9e2a472-a91f-402d-bf02-3459102e377f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr" Nov 02 20:31:19.370512 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.370473 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e2a472-a91f-402d-bf02-3459102e377f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a370c93c-52pkr\" (UID: \"c9e2a472-a91f-402d-bf02-3459102e377f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr" Nov 02 20:31:19.371176 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.371157 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e2a472-a91f-402d-bf02-3459102e377f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a370c93c-52pkr\" (UID: \"c9e2a472-a91f-402d-bf02-3459102e377f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr" Nov 02 20:31:19.372787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.372759 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e2a472-a91f-402d-bf02-3459102e377f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a370c93c-52pkr\" (UID: \"c9e2a472-a91f-402d-bf02-3459102e377f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr" Nov 02 20:31:19.428367 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.428339 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl7mf\" (UniqueName: \"kubernetes.io/projected/c9e2a472-a91f-402d-bf02-3459102e377f-kube-api-access-dl7mf\") pod \"maas-api-revoke-sub-a370c93c-52pkr\" (UID: \"c9e2a472-a91f-402d-bf02-3459102e377f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr" Nov 02 20:31:19.519189 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.519124 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr" Nov 02 20:31:19.914553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:19.914524 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:20.134160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:20.134129 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr"] Nov 02 20:31:20.135145 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:31:20.135121 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e2a472_a91f_402d_bf02_3459102e377f.slice/crio-35eaf8d46e171cbd772e216964b27da5c7a1b30283826f9812e0342a6314614e WatchSource:0}: Error finding container 35eaf8d46e171cbd772e216964b27da5c7a1b30283826f9812e0342a6314614e: Status 404 returned error can't find the container with id 35eaf8d46e171cbd772e216964b27da5c7a1b30283826f9812e0342a6314614e Nov 02 20:31:20.401548 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:20.401485 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr" event={"ID":"c9e2a472-a91f-402d-bf02-3459102e377f","Type":"ContainerStarted","Data":"35eaf8d46e171cbd772e216964b27da5c7a1b30283826f9812e0342a6314614e"} Nov 02 20:31:21.408742 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:21.408705 4971 generic.go:356] "Generic (PLEG): container finished" podID="c9e2a472-a91f-402d-bf02-3459102e377f" containerID="90d8c26a6aadbeb14c091babdfeb72a9114e14cdb0e66f1e134746625b83b8b2" exitCode=0 Nov 02 20:31:21.409156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:21.408793 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr" event={"ID":"c9e2a472-a91f-402d-bf02-3459102e377f","Type":"ContainerDied","Data":"90d8c26a6aadbeb14c091babdfeb72a9114e14cdb0e66f1e134746625b83b8b2"} Nov 02 20:31:22.236855 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.236821 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-gmvs9-mpwh2"] Nov 02 20:31:22.241286 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.241261 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" Nov 02 20:31:22.290446 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.290423 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9bjnj\"" Nov 02 20:31:22.293539 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.293507 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gmvs9-mpwh2"] Nov 02 20:31:22.332187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.332155 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c485622a-8d91-4523-bd63-97cce75df918-trusted-ca-bundle\") pod \"periodic-gathering-gmvs9-mpwh2\" (UID: \"c485622a-8d91-4523-bd63-97cce75df918\") " pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" Nov 02 20:31:22.332322 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.332284 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c485622a-8d91-4523-bd63-97cce75df918-archives-path\") pod \"periodic-gathering-gmvs9-mpwh2\" (UID: \"c485622a-8d91-4523-bd63-97cce75df918\") " pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" Nov 02 20:31:22.332376 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.332322 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c485622a-8d91-4523-bd63-97cce75df918-service-ca-bundle\") pod \"periodic-gathering-gmvs9-mpwh2\" (UID: \"c485622a-8d91-4523-bd63-97cce75df918\") " pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" Nov 02 20:31:22.332376 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.332344 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt2lf\" (UniqueName: \"kubernetes.io/projected/c485622a-8d91-4523-bd63-97cce75df918-kube-api-access-wt2lf\") pod \"periodic-gathering-gmvs9-mpwh2\" (UID: \"c485622a-8d91-4523-bd63-97cce75df918\") " pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" Nov 02 20:31:22.434432 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.434401 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c485622a-8d91-4523-bd63-97cce75df918-archives-path\") pod \"periodic-gathering-gmvs9-mpwh2\" (UID: \"c485622a-8d91-4523-bd63-97cce75df918\") " pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" Nov 02 20:31:22.434840 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.434461 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c485622a-8d91-4523-bd63-97cce75df918-service-ca-bundle\") pod \"periodic-gathering-gmvs9-mpwh2\" (UID: \"c485622a-8d91-4523-bd63-97cce75df918\") " pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" Nov 02 20:31:22.434840 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.434493 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt2lf\" (UniqueName: \"kubernetes.io/projected/c485622a-8d91-4523-bd63-97cce75df918-kube-api-access-wt2lf\") pod \"periodic-gathering-gmvs9-mpwh2\" (UID: \"c485622a-8d91-4523-bd63-97cce75df918\") " pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" Nov 02 20:31:22.434840 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.434614 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c485622a-8d91-4523-bd63-97cce75df918-trusted-ca-bundle\") pod \"periodic-gathering-gmvs9-mpwh2\" (UID: \"c485622a-8d91-4523-bd63-97cce75df918\") " pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" Nov 02 20:31:22.434840 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.434778 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c485622a-8d91-4523-bd63-97cce75df918-archives-path\") pod \"periodic-gathering-gmvs9-mpwh2\" (UID: \"c485622a-8d91-4523-bd63-97cce75df918\") " pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" Nov 02 20:31:22.435241 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.435220 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c485622a-8d91-4523-bd63-97cce75df918-service-ca-bundle\") pod \"periodic-gathering-gmvs9-mpwh2\" (UID: \"c485622a-8d91-4523-bd63-97cce75df918\") " pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" Nov 02 20:31:22.435624 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.435607 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c485622a-8d91-4523-bd63-97cce75df918-trusted-ca-bundle\") pod \"periodic-gathering-gmvs9-mpwh2\" (UID: \"c485622a-8d91-4523-bd63-97cce75df918\") " pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" Nov 02 20:31:22.492718 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.492657 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt2lf\" (UniqueName: \"kubernetes.io/projected/c485622a-8d91-4523-bd63-97cce75df918-kube-api-access-wt2lf\") pod \"periodic-gathering-gmvs9-mpwh2\" (UID: \"c485622a-8d91-4523-bd63-97cce75df918\") " pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" Nov 02 20:31:22.553733 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.553710 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" Nov 02 20:31:22.567439 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.567417 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr" Nov 02 20:31:22.640180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.639389 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e2a472-a91f-402d-bf02-3459102e377f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e2a472-a91f-402d-bf02-3459102e377f-maas-api-openshift-ca\") pod \"c9e2a472-a91f-402d-bf02-3459102e377f\" (UID: \"c9e2a472-a91f-402d-bf02-3459102e377f\") " Nov 02 20:31:22.640180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.639440 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e2a472-a91f-402d-bf02-3459102e377f-kube-api-access-dl7mf\" (UniqueName: \"kubernetes.io/projected/c9e2a472-a91f-402d-bf02-3459102e377f-kube-api-access-dl7mf\") pod \"c9e2a472-a91f-402d-bf02-3459102e377f\" (UID: \"c9e2a472-a91f-402d-bf02-3459102e377f\") " Nov 02 20:31:22.640180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.639591 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9e2a472-a91f-402d-bf02-3459102e377f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e2a472-a91f-402d-bf02-3459102e377f-maas-api-xks-ca\") pod \"c9e2a472-a91f-402d-bf02-3459102e377f\" (UID: \"c9e2a472-a91f-402d-bf02-3459102e377f\") " Nov 02 20:31:22.643844 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.643437 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e2a472-a91f-402d-bf02-3459102e377f-maas-api-openshift-ca" pod "c9e2a472-a91f-402d-bf02-3459102e377f" (UID: "c9e2a472-a91f-402d-bf02-3459102e377f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:22.644382 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.644285 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9e2a472-a91f-402d-bf02-3459102e377f-maas-api-xks-ca" pod "c9e2a472-a91f-402d-bf02-3459102e377f" (UID: "c9e2a472-a91f-402d-bf02-3459102e377f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:22.645974 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.645953 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e2a472-a91f-402d-bf02-3459102e377f-kube-api-access-dl7mf" pod "c9e2a472-a91f-402d-bf02-3459102e377f" (UID: "c9e2a472-a91f-402d-bf02-3459102e377f"). InnerVolumeSpecName "kube-api-access-dl7mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:22.740686 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.740644 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl7mf\" (UniqueName: \"kubernetes.io/projected/c9e2a472-a91f-402d-bf02-3459102e377f-kube-api-access-dl7mf\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:22.740686 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.740681 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e2a472-a91f-402d-bf02-3459102e377f-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:22.740686 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.740690 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e2a472-a91f-402d-bf02-3459102e377f-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:22.924148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:22.924021 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gmvs9-mpwh2"] Nov 02 20:31:22.924567 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:31:22.924537 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc485622a_8d91_4523_bd63_97cce75df918.slice/crio-04901f86469c7c4f8e6df45e3a3b769015dc450985b59c11ecd96699fd41e6e3 WatchSource:0}: Error finding container 04901f86469c7c4f8e6df45e3a3b769015dc450985b59c11ecd96699fd41e6e3: Status 404 returned error can't find the container with id 04901f86469c7c4f8e6df45e3a3b769015dc450985b59c11ecd96699fd41e6e3 Nov 02 20:31:23.424456 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:23.424431 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr" Nov 02 20:31:23.424624 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:23.424469 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr" event={"ID":"c9e2a472-a91f-402d-bf02-3459102e377f","Type":"ContainerDied","Data":"35eaf8d46e171cbd772e216964b27da5c7a1b30283826f9812e0342a6314614e"} Nov 02 20:31:23.424624 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:23.424514 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35eaf8d46e171cbd772e216964b27da5c7a1b30283826f9812e0342a6314614e" Nov 02 20:31:23.426292 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:23.426264 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" event={"ID":"c485622a-8d91-4523-bd63-97cce75df918","Type":"ContainerStarted","Data":"664a1332fc9c6dc9776fc2c9906697adb1fa7dd812397e3222db4cb2a9335b0d"} Nov 02 20:31:23.426401 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:23.426310 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" event={"ID":"c485622a-8d91-4523-bd63-97cce75df918","Type":"ContainerStarted","Data":"04901f86469c7c4f8e6df45e3a3b769015dc450985b59c11ecd96699fd41e6e3"} Nov 02 20:31:23.537804 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:23.537761 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" podStartSLOduration=1.5377476410000002 podStartE2EDuration="1.537747641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:23.534750948 +0000 UTC m=+1011.824407476" watchObservedRunningTime="2026-11-02 20:31:23.537747641 +0000 UTC m=+1011.827404170" Nov 02 20:31:24.712555 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:24.712518 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:27.330281 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:27.330246 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:27.596473 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:27.596430 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:29.032755 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.032717 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf"] Nov 02 20:31:29.033493 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.033472 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c9e2a472-a91f-402d-bf02-3459102e377f" containerName="revoke-keys" Nov 02 20:31:29.033493 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.033494 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e2a472-a91f-402d-bf02-3459102e377f" containerName="revoke-keys" Nov 02 20:31:29.033670 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.033656 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e2a472-a91f-402d-bf02-3459102e377f" containerName="revoke-keys" Nov 02 20:31:29.037782 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.037760 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" Nov 02 20:31:29.090197 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.090161 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf"] Nov 02 20:31:29.132220 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.132182 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkkj6\" (UniqueName: \"kubernetes.io/projected/e930cdea-4538-4672-a8a2-bf16d01ffb64-kube-api-access-jkkj6\") pod \"maas-api-revoke-sub-c209ba40-ln4bf\" (UID: \"e930cdea-4538-4672-a8a2-bf16d01ffb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" Nov 02 20:31:29.132437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.132420 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e930cdea-4538-4672-a8a2-bf16d01ffb64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c209ba40-ln4bf\" (UID: \"e930cdea-4538-4672-a8a2-bf16d01ffb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" Nov 02 20:31:29.132499 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.132452 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e930cdea-4538-4672-a8a2-bf16d01ffb64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c209ba40-ln4bf\" (UID: \"e930cdea-4538-4672-a8a2-bf16d01ffb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" Nov 02 20:31:29.133317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.133291 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:31:29.133474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.133322 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:31:29.233647 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.233613 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkkj6\" (UniqueName: \"kubernetes.io/projected/e930cdea-4538-4672-a8a2-bf16d01ffb64-kube-api-access-jkkj6\") pod \"maas-api-revoke-sub-c209ba40-ln4bf\" (UID: \"e930cdea-4538-4672-a8a2-bf16d01ffb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" Nov 02 20:31:29.233815 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.233723 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e930cdea-4538-4672-a8a2-bf16d01ffb64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c209ba40-ln4bf\" (UID: \"e930cdea-4538-4672-a8a2-bf16d01ffb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" Nov 02 20:31:29.233815 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.233750 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e930cdea-4538-4672-a8a2-bf16d01ffb64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c209ba40-ln4bf\" (UID: \"e930cdea-4538-4672-a8a2-bf16d01ffb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" Nov 02 20:31:29.234435 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.234407 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e930cdea-4538-4672-a8a2-bf16d01ffb64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c209ba40-ln4bf\" (UID: \"e930cdea-4538-4672-a8a2-bf16d01ffb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" Nov 02 20:31:29.236892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.236870 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e930cdea-4538-4672-a8a2-bf16d01ffb64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c209ba40-ln4bf\" (UID: \"e930cdea-4538-4672-a8a2-bf16d01ffb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" Nov 02 20:31:29.322336 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.322259 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkkj6\" (UniqueName: \"kubernetes.io/projected/e930cdea-4538-4672-a8a2-bf16d01ffb64-kube-api-access-jkkj6\") pod \"maas-api-revoke-sub-c209ba40-ln4bf\" (UID: \"e930cdea-4538-4672-a8a2-bf16d01ffb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" Nov 02 20:31:29.351921 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.351885 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" Nov 02 20:31:29.629673 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.629636 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:29.736664 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:29.736632 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf"] Nov 02 20:31:29.738415 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:31:29.738380 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode930cdea_4538_4672_a8a2_bf16d01ffb64.slice/crio-2e6ae96a792fb866d261761deb26286e68fb0c8406484588c29c2a17961be081 WatchSource:0}: Error finding container 2e6ae96a792fb866d261761deb26286e68fb0c8406484588c29c2a17961be081: Status 404 returned error can't find the container with id 2e6ae96a792fb866d261761deb26286e68fb0c8406484588c29c2a17961be081 Nov 02 20:31:30.473665 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:30.473630 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" event={"ID":"e930cdea-4538-4672-a8a2-bf16d01ffb64","Type":"ContainerStarted","Data":"5d4bea43888a31369f5bc36cdcab5e378f1cf25d9a4f16f28416e196ac6bb540"} Nov 02 20:31:30.474074 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:30.473679 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" event={"ID":"e930cdea-4538-4672-a8a2-bf16d01ffb64","Type":"ContainerStarted","Data":"2e6ae96a792fb866d261761deb26286e68fb0c8406484588c29c2a17961be081"} Nov 02 20:31:30.588665 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:30.588616 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" podStartSLOduration=2.588602533 podStartE2EDuration="2.588602533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:30.587516085 +0000 UTC m=+1018.877172616" watchObservedRunningTime="2026-11-02 20:31:30.588602533 +0000 UTC m=+1018.878259057" Nov 02 20:31:31.481127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:31.481076 4971 generic.go:356] "Generic (PLEG): container finished" podID="e930cdea-4538-4672-a8a2-bf16d01ffb64" containerID="5d4bea43888a31369f5bc36cdcab5e378f1cf25d9a4f16f28416e196ac6bb540" exitCode=0 Nov 02 20:31:31.481516 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:31.481174 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" event={"ID":"e930cdea-4538-4672-a8a2-bf16d01ffb64","Type":"ContainerDied","Data":"5d4bea43888a31369f5bc36cdcab5e378f1cf25d9a4f16f28416e196ac6bb540"} Nov 02 20:31:32.650532 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:32.650508 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" Nov 02 20:31:32.776421 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:32.776350 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e930cdea-4538-4672-a8a2-bf16d01ffb64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e930cdea-4538-4672-a8a2-bf16d01ffb64-maas-api-openshift-ca\") pod \"e930cdea-4538-4672-a8a2-bf16d01ffb64\" (UID: \"e930cdea-4538-4672-a8a2-bf16d01ffb64\") " Nov 02 20:31:32.776583 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:32.776441 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e930cdea-4538-4672-a8a2-bf16d01ffb64-kube-api-access-jkkj6\" (UniqueName: \"kubernetes.io/projected/e930cdea-4538-4672-a8a2-bf16d01ffb64-kube-api-access-jkkj6\") pod \"e930cdea-4538-4672-a8a2-bf16d01ffb64\" (UID: \"e930cdea-4538-4672-a8a2-bf16d01ffb64\") " Nov 02 20:31:32.776583 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:32.776551 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e930cdea-4538-4672-a8a2-bf16d01ffb64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e930cdea-4538-4672-a8a2-bf16d01ffb64-maas-api-xks-ca\") pod \"e930cdea-4538-4672-a8a2-bf16d01ffb64\" (UID: \"e930cdea-4538-4672-a8a2-bf16d01ffb64\") " Nov 02 20:31:32.776769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:32.776740 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e930cdea-4538-4672-a8a2-bf16d01ffb64-maas-api-openshift-ca" pod "e930cdea-4538-4672-a8a2-bf16d01ffb64" (UID: "e930cdea-4538-4672-a8a2-bf16d01ffb64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:32.777030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:32.776998 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e930cdea-4538-4672-a8a2-bf16d01ffb64-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:32.778973 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:32.778945 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e930cdea-4538-4672-a8a2-bf16d01ffb64-maas-api-xks-ca" pod "e930cdea-4538-4672-a8a2-bf16d01ffb64" (UID: "e930cdea-4538-4672-a8a2-bf16d01ffb64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:32.779269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:32.779245 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e930cdea-4538-4672-a8a2-bf16d01ffb64-kube-api-access-jkkj6" pod "e930cdea-4538-4672-a8a2-bf16d01ffb64" (UID: "e930cdea-4538-4672-a8a2-bf16d01ffb64"). InnerVolumeSpecName "kube-api-access-jkkj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:32.878520 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:32.878493 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e930cdea-4538-4672-a8a2-bf16d01ffb64-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:32.878520 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:32.878520 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkkj6\" (UniqueName: \"kubernetes.io/projected/e930cdea-4538-4672-a8a2-bf16d01ffb64-kube-api-access-jkkj6\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:33.337917 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:31:33.337882 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:31:33.494306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:33.494280 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" Nov 02 20:31:33.494447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:33.494276 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf" event={"ID":"e930cdea-4538-4672-a8a2-bf16d01ffb64","Type":"ContainerDied","Data":"2e6ae96a792fb866d261761deb26286e68fb0c8406484588c29c2a17961be081"} Nov 02 20:31:33.494447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:33.494418 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e6ae96a792fb866d261761deb26286e68fb0c8406484588c29c2a17961be081" Nov 02 20:31:34.415451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:34.415413 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:36.327108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.327057 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp"] Nov 02 20:31:36.327603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.327588 4971 cpu_manager.go:414] "containerMap: removing container" podUID="e930cdea-4538-4672-a8a2-bf16d01ffb64" containerName="revoke-keys" Nov 02 20:31:36.327643 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.327605 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e930cdea-4538-4672-a8a2-bf16d01ffb64" containerName="revoke-keys" Nov 02 20:31:36.327739 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.327730 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="e930cdea-4538-4672-a8a2-bf16d01ffb64" containerName="revoke-keys" Nov 02 20:31:36.331329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.331313 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" Nov 02 20:31:36.384611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.384571 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp"] Nov 02 20:31:36.413884 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.413841 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b28aad-566wp\" (UID: \"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" Nov 02 20:31:36.414095 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.413906 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b28aad-566wp\" (UID: \"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" Nov 02 20:31:36.414171 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.414115 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvgq\" (UniqueName: \"kubernetes.io/projected/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-kube-api-access-mmvgq\") pod \"maas-api-revoke-sub-a5b28aad-566wp\" (UID: \"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" Nov 02 20:31:36.427379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.427358 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:31:36.427481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.427464 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:31:36.515259 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.515225 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvgq\" (UniqueName: \"kubernetes.io/projected/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-kube-api-access-mmvgq\") pod \"maas-api-revoke-sub-a5b28aad-566wp\" (UID: \"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" Nov 02 20:31:36.515406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.515358 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b28aad-566wp\" (UID: \"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" Nov 02 20:31:36.515406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.515396 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b28aad-566wp\" (UID: \"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" Nov 02 20:31:36.515958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.515932 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b28aad-566wp\" (UID: \"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" Nov 02 20:31:36.517884 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.517862 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b28aad-566wp\" (UID: \"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" Nov 02 20:31:36.573702 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.573681 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvgq\" (UniqueName: \"kubernetes.io/projected/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-kube-api-access-mmvgq\") pod \"maas-api-revoke-sub-a5b28aad-566wp\" (UID: \"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" Nov 02 20:31:36.644535 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:36.644512 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" Nov 02 20:31:37.022997 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.022897 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp"] Nov 02 20:31:37.029402 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:31:37.029342 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2ed8b8_10c0_4e39_b6bc_1f999a5efc37.slice/crio-012eb64f368cd5a08186b8c15d0834ad3be7c1adb2b1628cc81043da035271ec WatchSource:0}: Error finding container 012eb64f368cd5a08186b8c15d0834ad3be7c1adb2b1628cc81043da035271ec: Status 404 returned error can't find the container with id 012eb64f368cd5a08186b8c15d0834ad3be7c1adb2b1628cc81043da035271ec Nov 02 20:31:37.223422 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.223390 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz"] Nov 02 20:31:37.229400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.229380 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" Nov 02 20:31:37.281224 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.281161 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz"] Nov 02 20:31:37.323851 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.323814 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vql8\" (UniqueName: \"kubernetes.io/projected/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-kube-api-access-2vql8\") pod \"maas-api-revoke-sub-94cee0f6-r6mgz\" (UID: \"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" Nov 02 20:31:37.323979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.323927 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94cee0f6-r6mgz\" (UID: \"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" Nov 02 20:31:37.323979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.323954 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94cee0f6-r6mgz\" (UID: \"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" Nov 02 20:31:37.425592 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.425564 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vql8\" (UniqueName: \"kubernetes.io/projected/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-kube-api-access-2vql8\") pod \"maas-api-revoke-sub-94cee0f6-r6mgz\" (UID: \"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" Nov 02 20:31:37.426065 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.425635 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94cee0f6-r6mgz\" (UID: \"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" Nov 02 20:31:37.426065 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.425653 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94cee0f6-r6mgz\" (UID: \"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" Nov 02 20:31:37.426427 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.426404 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94cee0f6-r6mgz\" (UID: \"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" Nov 02 20:31:37.428141 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.428123 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94cee0f6-r6mgz\" (UID: \"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" Nov 02 20:31:37.483684 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.483663 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vql8\" (UniqueName: \"kubernetes.io/projected/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-kube-api-access-2vql8\") pod \"maas-api-revoke-sub-94cee0f6-r6mgz\" (UID: \"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" Nov 02 20:31:37.519095 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.519055 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" event={"ID":"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37","Type":"ContainerStarted","Data":"42913c6414652902ab20ab5023e1a07a9c88ffb52089071d604571fe3a5f32a2"} Nov 02 20:31:37.519184 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.519138 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" event={"ID":"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37","Type":"ContainerStarted","Data":"012eb64f368cd5a08186b8c15d0834ad3be7c1adb2b1628cc81043da035271ec"} Nov 02 20:31:37.556068 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.556022 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" Nov 02 20:31:37.631729 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.631677 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" podStartSLOduration=1.631662744 podStartE2EDuration="1.631662744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:37.628605162 +0000 UTC m=+1025.918261691" watchObservedRunningTime="2026-11-02 20:31:37.631662744 +0000 UTC m=+1025.921319327" Nov 02 20:31:37.926839 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:37.926813 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz"] Nov 02 20:31:38.386130 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:38.386096 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk"] Nov 02 20:31:38.386471 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:38.386434 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" podUID="4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" containerName="istio-proxy" containerID="cri-o://830e90b89c9d409b8dcdb929c52925940cbce950f2734dccedf5a4b02824c8cf" gracePeriod=30 Nov 02 20:31:38.527733 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:38.527653 4971 generic.go:356] "Generic (PLEG): container finished" podID="1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37" containerID="42913c6414652902ab20ab5023e1a07a9c88ffb52089071d604571fe3a5f32a2" exitCode=0 Nov 02 20:31:38.528129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:38.527762 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" event={"ID":"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37","Type":"ContainerDied","Data":"42913c6414652902ab20ab5023e1a07a9c88ffb52089071d604571fe3a5f32a2"} Nov 02 20:31:38.529718 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:38.529689 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" event={"ID":"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4","Type":"ContainerStarted","Data":"098172e609de205aeed162d6576008e027cf36281ea4810cee3411a1e4307cee"} Nov 02 20:31:38.529853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:38.529738 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" event={"ID":"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4","Type":"ContainerStarted","Data":"223caeb388b493294c8f42e89df1f776a2f2f9ef987f8390a6ea04b7da8846b0"} Nov 02 20:31:38.815139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:38.815016 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" podStartSLOduration=1.8150030639999999 podStartE2EDuration="1.815003064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:38.812812185 +0000 UTC m=+1027.102468714" watchObservedRunningTime="2026-11-02 20:31:38.815003064 +0000 UTC m=+1027.104659591" Nov 02 20:31:39.255620 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.255589 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:39.345623 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.345594 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4"] Nov 02 20:31:39.356010 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.355973 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" Nov 02 20:31:39.404961 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.404936 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4"] Nov 02 20:31:39.456949 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.456919 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc8a70c-008d-4aba-878c-4f533ccd1930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d64601e-cvsp4\" (UID: \"7bc8a70c-008d-4aba-878c-4f533ccd1930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" Nov 02 20:31:39.457148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.456979 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc8a70c-008d-4aba-878c-4f533ccd1930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d64601e-cvsp4\" (UID: \"7bc8a70c-008d-4aba-878c-4f533ccd1930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" Nov 02 20:31:39.457240 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.457214 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz7q5\" (UniqueName: \"kubernetes.io/projected/7bc8a70c-008d-4aba-878c-4f533ccd1930-kube-api-access-hz7q5\") pod \"maas-api-revoke-sub-0d64601e-cvsp4\" (UID: \"7bc8a70c-008d-4aba-878c-4f533ccd1930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" Nov 02 20:31:39.537748 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.537664 4971 generic.go:356] "Generic (PLEG): container finished" podID="83fd427a-ec1c-4eb0-8b0d-73a49c0174e4" containerID="098172e609de205aeed162d6576008e027cf36281ea4810cee3411a1e4307cee" exitCode=0 Nov 02 20:31:39.537748 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.537732 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" event={"ID":"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4","Type":"ContainerDied","Data":"098172e609de205aeed162d6576008e027cf36281ea4810cee3411a1e4307cee"} Nov 02 20:31:39.558845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.558812 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc8a70c-008d-4aba-878c-4f533ccd1930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d64601e-cvsp4\" (UID: \"7bc8a70c-008d-4aba-878c-4f533ccd1930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" Nov 02 20:31:39.558986 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.558902 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc8a70c-008d-4aba-878c-4f533ccd1930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d64601e-cvsp4\" (UID: \"7bc8a70c-008d-4aba-878c-4f533ccd1930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" Nov 02 20:31:39.559062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.559019 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz7q5\" (UniqueName: \"kubernetes.io/projected/7bc8a70c-008d-4aba-878c-4f533ccd1930-kube-api-access-hz7q5\") pod \"maas-api-revoke-sub-0d64601e-cvsp4\" (UID: \"7bc8a70c-008d-4aba-878c-4f533ccd1930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" Nov 02 20:31:39.559574 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.559555 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc8a70c-008d-4aba-878c-4f533ccd1930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d64601e-cvsp4\" (UID: \"7bc8a70c-008d-4aba-878c-4f533ccd1930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" Nov 02 20:31:39.561481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.561461 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc8a70c-008d-4aba-878c-4f533ccd1930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d64601e-cvsp4\" (UID: \"7bc8a70c-008d-4aba-878c-4f533ccd1930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" Nov 02 20:31:39.617719 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.617688 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz7q5\" (UniqueName: \"kubernetes.io/projected/7bc8a70c-008d-4aba-878c-4f533ccd1930-kube-api-access-hz7q5\") pod \"maas-api-revoke-sub-0d64601e-cvsp4\" (UID: \"7bc8a70c-008d-4aba-878c-4f533ccd1930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" Nov 02 20:31:39.669918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.669893 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" Nov 02 20:31:39.712323 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.712296 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" Nov 02 20:31:39.861901 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.861875 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-maas-api-openshift-ca\") pod \"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37\" (UID: \"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37\") " Nov 02 20:31:39.862068 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.861938 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-kube-api-access-mmvgq\" (UniqueName: \"kubernetes.io/projected/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-kube-api-access-mmvgq\") pod \"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37\" (UID: \"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37\") " Nov 02 20:31:39.862157 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.862071 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-maas-api-xks-ca\") pod \"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37\" (UID: \"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37\") " Nov 02 20:31:39.862317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.862285 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-maas-api-openshift-ca" pod "1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37" (UID: "1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:39.862530 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.862513 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:39.864225 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.864204 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-maas-api-xks-ca" pod "1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37" (UID: "1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:39.864527 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.864506 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-kube-api-access-mmvgq" pod "1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37" (UID: "1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37"). InnerVolumeSpecName "kube-api-access-mmvgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:39.963728 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.963701 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:39.963728 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:39.963725 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmvgq\" (UniqueName: \"kubernetes.io/projected/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37-kube-api-access-mmvgq\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:40.044663 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:31:40.044620 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc8a70c_008d_4aba_878c_4f533ccd1930.slice/crio-b7fdfd7678c553dad46b9d72e896bfb35ac3d913400acb2ac029e7b4ac2e2daf WatchSource:0}: Error finding container b7fdfd7678c553dad46b9d72e896bfb35ac3d913400acb2ac029e7b4ac2e2daf: Status 404 returned error can't find the container with id b7fdfd7678c553dad46b9d72e896bfb35ac3d913400acb2ac029e7b4ac2e2daf Nov 02 20:31:40.045907 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.045882 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4"] Nov 02 20:31:40.545401 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.545333 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" Nov 02 20:31:40.545401 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.545376 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp" event={"ID":"1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37","Type":"ContainerDied","Data":"012eb64f368cd5a08186b8c15d0834ad3be7c1adb2b1628cc81043da035271ec"} Nov 02 20:31:40.545808 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.545425 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="012eb64f368cd5a08186b8c15d0834ad3be7c1adb2b1628cc81043da035271ec" Nov 02 20:31:40.547163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.547137 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" event={"ID":"7bc8a70c-008d-4aba-878c-4f533ccd1930","Type":"ContainerStarted","Data":"be25fbc007273fbebb1fc46f992e00cb5f3e6b199ddb724305dd012749b4a1c4"} Nov 02 20:31:40.547270 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.547186 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" event={"ID":"7bc8a70c-008d-4aba-878c-4f533ccd1930","Type":"ContainerStarted","Data":"b7fdfd7678c553dad46b9d72e896bfb35ac3d913400acb2ac029e7b4ac2e2daf"} Nov 02 20:31:40.659416 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.659360 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" podStartSLOduration=1.6593462410000002 podStartE2EDuration="1.659346241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:40.656651486 +0000 UTC m=+1028.946308015" watchObservedRunningTime="2026-11-02 20:31:40.659346241 +0000 UTC m=+1028.949002809" Nov 02 20:31:40.710043 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.710019 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" Nov 02 20:31:40.872444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.872421 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-maas-api-xks-ca\") pod \"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4\" (UID: \"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4\") " Nov 02 20:31:40.872613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.872497 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-kube-api-access-2vql8\" (UniqueName: \"kubernetes.io/projected/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-kube-api-access-2vql8\") pod \"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4\" (UID: \"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4\") " Nov 02 20:31:40.872613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.872567 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-maas-api-openshift-ca\") pod \"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4\" (UID: \"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4\") " Nov 02 20:31:40.872885 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.872858 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-maas-api-openshift-ca" pod "83fd427a-ec1c-4eb0-8b0d-73a49c0174e4" (UID: "83fd427a-ec1c-4eb0-8b0d-73a49c0174e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:40.874790 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.874767 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-kube-api-access-2vql8" pod "83fd427a-ec1c-4eb0-8b0d-73a49c0174e4" (UID: "83fd427a-ec1c-4eb0-8b0d-73a49c0174e4"). InnerVolumeSpecName "kube-api-access-2vql8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:40.874854 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.874815 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-maas-api-xks-ca" pod "83fd427a-ec1c-4eb0-8b0d-73a49c0174e4" (UID: "83fd427a-ec1c-4eb0-8b0d-73a49c0174e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:40.974404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.974370 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vql8\" (UniqueName: \"kubernetes.io/projected/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-kube-api-access-2vql8\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:40.974404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.974404 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:40.974557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:40.974420 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:41.554843 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:41.554815 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" Nov 02 20:31:41.555299 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:41.554821 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz" event={"ID":"83fd427a-ec1c-4eb0-8b0d-73a49c0174e4","Type":"ContainerDied","Data":"223caeb388b493294c8f42e89df1f776a2f2f9ef987f8390a6ea04b7da8846b0"} Nov 02 20:31:41.555299 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:41.554941 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="223caeb388b493294c8f42e89df1f776a2f2f9ef987f8390a6ea04b7da8846b0" Nov 02 20:31:41.556509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:41.556482 4971 generic.go:356] "Generic (PLEG): container finished" podID="7bc8a70c-008d-4aba-878c-4f533ccd1930" containerID="be25fbc007273fbebb1fc46f992e00cb5f3e6b199ddb724305dd012749b4a1c4" exitCode=0 Nov 02 20:31:41.556628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:41.556546 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" event={"ID":"7bc8a70c-008d-4aba-878c-4f533ccd1930","Type":"ContainerDied","Data":"be25fbc007273fbebb1fc46f992e00cb5f3e6b199ddb724305dd012749b4a1c4"} Nov 02 20:31:42.730886 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:42.730858 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" Nov 02 20:31:42.896552 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:42.896523 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc8a70c-008d-4aba-878c-4f533ccd1930-kube-api-access-hz7q5\" (UniqueName: \"kubernetes.io/projected/7bc8a70c-008d-4aba-878c-4f533ccd1930-kube-api-access-hz7q5\") pod \"7bc8a70c-008d-4aba-878c-4f533ccd1930\" (UID: \"7bc8a70c-008d-4aba-878c-4f533ccd1930\") " Nov 02 20:31:42.896720 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:42.896593 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bc8a70c-008d-4aba-878c-4f533ccd1930-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc8a70c-008d-4aba-878c-4f533ccd1930-maas-api-openshift-ca\") pod \"7bc8a70c-008d-4aba-878c-4f533ccd1930\" (UID: \"7bc8a70c-008d-4aba-878c-4f533ccd1930\") " Nov 02 20:31:42.896720 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:42.896629 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bc8a70c-008d-4aba-878c-4f533ccd1930-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc8a70c-008d-4aba-878c-4f533ccd1930-maas-api-xks-ca\") pod \"7bc8a70c-008d-4aba-878c-4f533ccd1930\" (UID: \"7bc8a70c-008d-4aba-878c-4f533ccd1930\") " Nov 02 20:31:42.896946 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:42.896925 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc8a70c-008d-4aba-878c-4f533ccd1930-maas-api-openshift-ca" pod "7bc8a70c-008d-4aba-878c-4f533ccd1930" (UID: "7bc8a70c-008d-4aba-878c-4f533ccd1930"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:42.897051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:42.897032 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc8a70c-008d-4aba-878c-4f533ccd1930-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:42.898723 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:42.898701 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc8a70c-008d-4aba-878c-4f533ccd1930-maas-api-xks-ca" pod "7bc8a70c-008d-4aba-878c-4f533ccd1930" (UID: "7bc8a70c-008d-4aba-878c-4f533ccd1930"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:42.898830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:42.898817 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc8a70c-008d-4aba-878c-4f533ccd1930-kube-api-access-hz7q5" pod "7bc8a70c-008d-4aba-878c-4f533ccd1930" (UID: "7bc8a70c-008d-4aba-878c-4f533ccd1930"). InnerVolumeSpecName "kube-api-access-hz7q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:42.998250 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:42.998224 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz7q5\" (UniqueName: \"kubernetes.io/projected/7bc8a70c-008d-4aba-878c-4f533ccd1930-kube-api-access-hz7q5\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:42.998250 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:42.998248 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc8a70c-008d-4aba-878c-4f533ccd1930-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.575007 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.574979 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" Nov 02 20:31:43.575166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.574976 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4" event={"ID":"7bc8a70c-008d-4aba-878c-4f533ccd1930","Type":"ContainerDied","Data":"b7fdfd7678c553dad46b9d72e896bfb35ac3d913400acb2ac029e7b4ac2e2daf"} Nov 02 20:31:43.575166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.575143 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7fdfd7678c553dad46b9d72e896bfb35ac3d913400acb2ac029e7b4ac2e2daf" Nov 02 20:31:43.577450 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.577419 4971 generic.go:356] "Generic (PLEG): container finished" podID="4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" containerID="830e90b89c9d409b8dcdb929c52925940cbce950f2734dccedf5a4b02824c8cf" exitCode=0 Nov 02 20:31:43.577589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.577478 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" event={"ID":"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9","Type":"ContainerDied","Data":"830e90b89c9d409b8dcdb929c52925940cbce950f2734dccedf5a4b02824c8cf"} Nov 02 20:31:43.630886 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.630863 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:31:43.814573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.814542 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-token\" (UniqueName: \"kubernetes.io/projected/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-token\") pod \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " Nov 02 20:31:43.814985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.814653 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-workload-socket\") pod \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " Nov 02 20:31:43.814985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.814692 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-workload-certs\") pod \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " Nov 02 20:31:43.814985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.814719 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-podinfo\") pod \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " Nov 02 20:31:43.814985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.814768 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-credential-socket\") pod \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " Nov 02 20:31:43.814985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.814800 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-kube-api-access-tc5m7\" (UniqueName: \"kubernetes.io/projected/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-kube-api-access-tc5m7\") pod \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " Nov 02 20:31:43.814985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.814891 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-envoy\") pod \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " Nov 02 20:31:43.814985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.814983 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-data\") pod \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " Nov 02 20:31:43.815369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.815000 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-workload-socket" pod "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" (UID: "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:43.815369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.815012 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-workload-certs" pod "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" (UID: "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:43.815369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.815076 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istiod-ca-cert\") pod \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\" (UID: \"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9\") " Nov 02 20:31:43.815369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.815216 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-credential-socket" pod "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" (UID: "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:43.815369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.815243 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-data" pod "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" (UID: "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:43.815566 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.815538 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istiod-ca-cert" pod "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" (UID: "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:43.815648 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.815629 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.815691 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.815654 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.815691 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.815669 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.815691 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.815683 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.817366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.817340 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-token" pod "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" (UID: "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:43.817652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.817633 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-kube-api-access-tc5m7" pod "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" (UID: "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9"). InnerVolumeSpecName "kube-api-access-tc5m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:43.817769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.817744 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-envoy" pod "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" (UID: "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:43.817878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.817818 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-podinfo" pod "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" (UID: "4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:31:43.916865 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.916827 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.916865 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.916860 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.917037 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.916874 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.917037 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.916887 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc5m7\" (UniqueName: \"kubernetes.io/projected/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-kube-api-access-tc5m7\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.917037 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:43.916900 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:44.106603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:44.106571 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:44.585465 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:44.585386 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" Nov 02 20:31:44.585465 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:44.585396 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk" event={"ID":"4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9","Type":"ContainerDied","Data":"a6cdfc5bfc66356913ba28baeccc0cd99191c96ee5c5a9facc6a27cbf25423ff"} Nov 02 20:31:44.585465 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:44.585464 4971 scope.go:122] "RemoveContainer" containerID="830e90b89c9d409b8dcdb929c52925940cbce950f2734dccedf5a4b02824c8cf" Nov 02 20:31:44.742021 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:44.741985 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk"] Nov 02 20:31:44.788624 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:44.788601 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-k98hk"] Nov 02 20:31:45.866388 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:45.866356 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" path="/var/lib/kubelet/pods/4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9/volumes" Nov 02 20:31:47.370328 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.370296 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz"] Nov 02 20:31:47.370993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.370974 4971 cpu_manager.go:414] "containerMap: removing container" podUID="4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" containerName="istio-proxy" Nov 02 20:31:47.371074 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.370997 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" containerName="istio-proxy" Nov 02 20:31:47.371074 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.371022 4971 cpu_manager.go:414] "containerMap: removing container" podUID="1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37" containerName="revoke-keys" Nov 02 20:31:47.371074 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.371027 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37" containerName="revoke-keys" Nov 02 20:31:47.371074 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.371049 4971 cpu_manager.go:414] "containerMap: removing container" podUID="83fd427a-ec1c-4eb0-8b0d-73a49c0174e4" containerName="revoke-keys" Nov 02 20:31:47.371074 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.371055 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83fd427a-ec1c-4eb0-8b0d-73a49c0174e4" containerName="revoke-keys" Nov 02 20:31:47.371260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.371095 4971 cpu_manager.go:414] "containerMap: removing container" podUID="7bc8a70c-008d-4aba-878c-4f533ccd1930" containerName="revoke-keys" Nov 02 20:31:47.371260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.371102 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc8a70c-008d-4aba-878c-4f533ccd1930" containerName="revoke-keys" Nov 02 20:31:47.371260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.371215 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="83fd427a-ec1c-4eb0-8b0d-73a49c0174e4" containerName="revoke-keys" Nov 02 20:31:47.371260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.371231 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc8a70c-008d-4aba-878c-4f533ccd1930" containerName="revoke-keys" Nov 02 20:31:47.371260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.371238 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="4071971f-4b3b-45f8-9ba4-ea8eb9f6e1b9" containerName="istio-proxy" Nov 02 20:31:47.371260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.371245 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37" containerName="revoke-keys" Nov 02 20:31:47.374661 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.374645 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz" Nov 02 20:31:47.432338 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.432312 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz"] Nov 02 20:31:47.475270 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.475246 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:31:47.475498 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.475482 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:31:47.555444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.555417 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hqks\" (UniqueName: \"kubernetes.io/projected/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-kube-api-access-5hqks\") pod \"maas-api-revoke-sub-288cc6ec-gpxwz\" (UID: \"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz" Nov 02 20:31:47.555575 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.555474 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-288cc6ec-gpxwz\" (UID: \"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz" Nov 02 20:31:47.555637 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.555618 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-288cc6ec-gpxwz\" (UID: \"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz" Nov 02 20:31:47.582719 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.582696 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb"] Nov 02 20:31:47.586625 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.586609 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" Nov 02 20:31:47.640627 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.640563 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb"] Nov 02 20:31:47.656160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.656128 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-288cc6ec-gpxwz\" (UID: \"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz" Nov 02 20:31:47.656267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.656166 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6725\" (UniqueName: \"kubernetes.io/projected/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-kube-api-access-h6725\") pod \"maas-api-revoke-sub-67dbdfb1-skpfb\" (UID: \"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" Nov 02 20:31:47.656267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.656220 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hqks\" (UniqueName: \"kubernetes.io/projected/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-kube-api-access-5hqks\") pod \"maas-api-revoke-sub-288cc6ec-gpxwz\" (UID: \"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz" Nov 02 20:31:47.656361 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.656269 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-288cc6ec-gpxwz\" (UID: \"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz" Nov 02 20:31:47.656361 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.656328 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67dbdfb1-skpfb\" (UID: \"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" Nov 02 20:31:47.656361 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.656354 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67dbdfb1-skpfb\" (UID: \"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" Nov 02 20:31:47.656941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.656924 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-288cc6ec-gpxwz\" (UID: \"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz" Nov 02 20:31:47.658809 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.658781 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-288cc6ec-gpxwz\" (UID: \"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz" Nov 02 20:31:47.714312 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.714290 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hqks\" (UniqueName: \"kubernetes.io/projected/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-kube-api-access-5hqks\") pod \"maas-api-revoke-sub-288cc6ec-gpxwz\" (UID: \"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz" Nov 02 20:31:47.756901 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.756873 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6725\" (UniqueName: \"kubernetes.io/projected/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-kube-api-access-h6725\") pod \"maas-api-revoke-sub-67dbdfb1-skpfb\" (UID: \"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" Nov 02 20:31:47.757022 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.757007 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67dbdfb1-skpfb\" (UID: \"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" Nov 02 20:31:47.757075 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.757034 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67dbdfb1-skpfb\" (UID: \"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" Nov 02 20:31:47.757787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.757762 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67dbdfb1-skpfb\" (UID: \"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" Nov 02 20:31:47.759894 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.759857 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67dbdfb1-skpfb\" (UID: \"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" Nov 02 20:31:47.815483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.815460 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6725\" (UniqueName: \"kubernetes.io/projected/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-kube-api-access-h6725\") pod \"maas-api-revoke-sub-67dbdfb1-skpfb\" (UID: \"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" Nov 02 20:31:47.900374 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.900317 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" Nov 02 20:31:47.988441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:47.988415 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz" Nov 02 20:31:48.270341 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:48.270316 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb"] Nov 02 20:31:48.270587 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:31:48.270563 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52a6ba6_42e7_4cb8_b04d_9f50939de2a5.slice/crio-e663027088ea634803bb536527301292449b892948c4fe7ef43a10e566d92b5e WatchSource:0}: Error finding container e663027088ea634803bb536527301292449b892948c4fe7ef43a10e566d92b5e: Status 404 returned error can't find the container with id e663027088ea634803bb536527301292449b892948c4fe7ef43a10e566d92b5e Nov 02 20:31:48.367259 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:31:48.367218 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa2bd90_0708_4c95_aa5d_9df2b3d1965e.slice/crio-f245bc7f5a5d122d0b735fa99730ebe8ee245e1b2f3b9dfff0f0595598139bab WatchSource:0}: Error finding container f245bc7f5a5d122d0b735fa99730ebe8ee245e1b2f3b9dfff0f0595598139bab: Status 404 returned error can't find the container with id f245bc7f5a5d122d0b735fa99730ebe8ee245e1b2f3b9dfff0f0595598139bab Nov 02 20:31:48.367629 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:48.367601 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz"] Nov 02 20:31:48.612531 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:48.612490 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" event={"ID":"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5","Type":"ContainerStarted","Data":"134b3317fb144f62b37cab54c461c5efa0fa34ebc0e5de3f41f9feb6416b2472"} Nov 02 20:31:48.612883 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:48.612552 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" event={"ID":"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5","Type":"ContainerStarted","Data":"e663027088ea634803bb536527301292449b892948c4fe7ef43a10e566d92b5e"} Nov 02 20:31:48.613915 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:48.613893 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz" event={"ID":"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e","Type":"ContainerStarted","Data":"f245bc7f5a5d122d0b735fa99730ebe8ee245e1b2f3b9dfff0f0595598139bab"} Nov 02 20:31:48.723206 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:48.723160 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" podStartSLOduration=1.723146898 podStartE2EDuration="1.723146898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:48.721579116 +0000 UTC m=+1037.011235643" watchObservedRunningTime="2026-11-02 20:31:48.723146898 +0000 UTC m=+1037.012803426" Nov 02 20:31:49.036530 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.036499 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:49.625333 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.625300 4971 generic.go:356] "Generic (PLEG): container finished" podID="cfa2bd90-0708-4c95-aa5d-9df2b3d1965e" containerID="b2ee1a5b67dcce77bda867659c3fb57b3177894b4961392b92539f1ef7585335" exitCode=0 Nov 02 20:31:49.625766 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.625368 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz" event={"ID":"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e","Type":"ContainerDied","Data":"b2ee1a5b67dcce77bda867659c3fb57b3177894b4961392b92539f1ef7585335"} Nov 02 20:31:49.627349 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.627311 4971 generic.go:356] "Generic (PLEG): container finished" podID="d52a6ba6-42e7-4cb8-b04d-9f50939de2a5" containerID="134b3317fb144f62b37cab54c461c5efa0fa34ebc0e5de3f41f9feb6416b2472" exitCode=0 Nov 02 20:31:49.627471 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.627368 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" event={"ID":"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5","Type":"ContainerDied","Data":"134b3317fb144f62b37cab54c461c5efa0fa34ebc0e5de3f41f9feb6416b2472"} Nov 02 20:31:49.638311 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.638281 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6"] Nov 02 20:31:49.642340 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.642321 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.680460 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.680429 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.680571 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.680471 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.680690 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.680671 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.680756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.680720 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebf56dbf-ece9-4da3-9589-77c972d8552e-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.680798 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.680782 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdwz4\" (UniqueName: \"kubernetes.io/projected/ebf56dbf-ece9-4da3-9589-77c972d8552e-kube-api-access-wdwz4\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.680840 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.680818 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.680888 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.680840 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.680888 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.680861 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.680976 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.680963 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.691793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.691775 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-bce768-openshift-default-dockercfg-xdqwh\"" Nov 02 20:31:49.737589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.737562 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6"] Nov 02 20:31:49.782403 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.782366 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.782541 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.782409 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebf56dbf-ece9-4da3-9589-77c972d8552e-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.782541 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.782428 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdwz4\" (UniqueName: \"kubernetes.io/projected/ebf56dbf-ece9-4da3-9589-77c972d8552e-kube-api-access-wdwz4\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.782541 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.782451 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.782541 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.782476 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.782541 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.782510 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.782841 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.782577 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.782841 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.782641 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.782841 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.782682 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.783139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.783049 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.783256 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.783116 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.783343 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.783316 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.783435 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.783415 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.783493 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.783446 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebf56dbf-ece9-4da3-9589-77c972d8552e-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.784907 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.784887 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.785463 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.785446 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.840419 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.840390 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.840537 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.840458 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdwz4\" (UniqueName: \"kubernetes.io/projected/ebf56dbf-ece9-4da3-9589-77c972d8552e-kube-api-access-wdwz4\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:49.955471 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:49.955396 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:50.142305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.142276 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw"] Nov 02 20:31:50.148167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.148144 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" Nov 02 20:31:50.186471 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.186445 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7baadcc3-0541-424e-9bc1-c28eb755787a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-419a3148-kx9dw\" (UID: \"7baadcc3-0541-424e-9bc1-c28eb755787a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" Nov 02 20:31:50.186620 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.186486 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7baadcc3-0541-424e-9bc1-c28eb755787a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-419a3148-kx9dw\" (UID: \"7baadcc3-0541-424e-9bc1-c28eb755787a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" Nov 02 20:31:50.186662 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.186633 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55kjj\" (UniqueName: \"kubernetes.io/projected/7baadcc3-0541-424e-9bc1-c28eb755787a-kube-api-access-55kjj\") pod \"maas-api-revoke-sub-419a3148-kx9dw\" (UID: \"7baadcc3-0541-424e-9bc1-c28eb755787a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" Nov 02 20:31:50.201255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.201226 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw"] Nov 02 20:31:50.287624 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.287531 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7baadcc3-0541-424e-9bc1-c28eb755787a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-419a3148-kx9dw\" (UID: \"7baadcc3-0541-424e-9bc1-c28eb755787a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" Nov 02 20:31:50.287797 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.287656 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55kjj\" (UniqueName: \"kubernetes.io/projected/7baadcc3-0541-424e-9bc1-c28eb755787a-kube-api-access-55kjj\") pod \"maas-api-revoke-sub-419a3148-kx9dw\" (UID: \"7baadcc3-0541-424e-9bc1-c28eb755787a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" Nov 02 20:31:50.287883 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.287860 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7baadcc3-0541-424e-9bc1-c28eb755787a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-419a3148-kx9dw\" (UID: \"7baadcc3-0541-424e-9bc1-c28eb755787a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" Nov 02 20:31:50.288391 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.288365 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7baadcc3-0541-424e-9bc1-c28eb755787a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-419a3148-kx9dw\" (UID: \"7baadcc3-0541-424e-9bc1-c28eb755787a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" Nov 02 20:31:50.290827 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.290555 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7baadcc3-0541-424e-9bc1-c28eb755787a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-419a3148-kx9dw\" (UID: \"7baadcc3-0541-424e-9bc1-c28eb755787a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" Nov 02 20:31:50.345766 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.345738 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55kjj\" (UniqueName: \"kubernetes.io/projected/7baadcc3-0541-424e-9bc1-c28eb755787a-kube-api-access-55kjj\") pod \"maas-api-revoke-sub-419a3148-kx9dw\" (UID: \"7baadcc3-0541-424e-9bc1-c28eb755787a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" Nov 02 20:31:50.393003 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:31:50.392970 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf56dbf_ece9_4da3_9589_77c972d8552e.slice/crio-6d2531efca5d0c2af972a8ec401ce1f5df93299d4ed85a48184db03d276652ec WatchSource:0}: Error finding container 6d2531efca5d0c2af972a8ec401ce1f5df93299d4ed85a48184db03d276652ec: Status 404 returned error can't find the container with id 6d2531efca5d0c2af972a8ec401ce1f5df93299d4ed85a48184db03d276652ec Nov 02 20:31:50.393296 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.393272 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6"] Nov 02 20:31:50.395474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.395445 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:31:50.395563 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.395516 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:31:50.395563 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.395558 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:31:50.464535 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.464507 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" Nov 02 20:31:50.639626 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.639152 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" event={"ID":"ebf56dbf-ece9-4da3-9589-77c972d8552e","Type":"ContainerStarted","Data":"3e386a4b3a9b03dc05d87e737a9b9b44130d8cf45f07328608b3a081d5b2763b"} Nov 02 20:31:50.639626 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.639213 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" event={"ID":"ebf56dbf-ece9-4da3-9589-77c972d8552e","Type":"ContainerStarted","Data":"6d2531efca5d0c2af972a8ec401ce1f5df93299d4ed85a48184db03d276652ec"} Nov 02 20:31:50.761538 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.755648 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" podStartSLOduration=1.7556250169999998 podStartE2EDuration="1.755625017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:50.754501975 +0000 UTC m=+1039.044158503" watchObservedRunningTime="2026-11-02 20:31:50.755625017 +0000 UTC m=+1039.045281546" Nov 02 20:31:50.865344 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.865316 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw"] Nov 02 20:31:50.865473 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:31:50.865446 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7baadcc3_0541_424e_9bc1_c28eb755787a.slice/crio-ba66043e1a83d48a99c28d1c834b3ee2ebe1c04bbb6dd63a35108bcb9822b46d WatchSource:0}: Error finding container ba66043e1a83d48a99c28d1c834b3ee2ebe1c04bbb6dd63a35108bcb9822b46d: Status 404 returned error can't find the container with id ba66043e1a83d48a99c28d1c834b3ee2ebe1c04bbb6dd63a35108bcb9822b46d Nov 02 20:31:50.883046 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.883024 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" Nov 02 20:31:50.886280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.886254 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz" Nov 02 20:31:50.955830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:50.955800 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:51.006606 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.006481 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-kube-api-access-5hqks\" (UniqueName: \"kubernetes.io/projected/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-kube-api-access-5hqks\") pod \"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e\" (UID: \"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e\") " Nov 02 20:31:51.006754 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.006662 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-maas-api-openshift-ca\") pod \"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e\" (UID: \"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e\") " Nov 02 20:31:51.006754 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.006723 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-kube-api-access-h6725\" (UniqueName: \"kubernetes.io/projected/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-kube-api-access-h6725\") pod \"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5\" (UID: \"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5\") " Nov 02 20:31:51.006830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.006773 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-maas-api-xks-ca\") pod \"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e\" (UID: \"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e\") " Nov 02 20:31:51.006830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.006812 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-maas-api-openshift-ca\") pod \"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5\" (UID: \"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5\") " Nov 02 20:31:51.006895 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.006845 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-maas-api-xks-ca\") pod \"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5\" (UID: \"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5\") " Nov 02 20:31:51.010157 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.008230 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-maas-api-openshift-ca" pod "cfa2bd90-0708-4c95-aa5d-9df2b3d1965e" (UID: "cfa2bd90-0708-4c95-aa5d-9df2b3d1965e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:51.010157 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.008459 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-maas-api-openshift-ca" pod "d52a6ba6-42e7-4cb8-b04d-9f50939de2a5" (UID: "d52a6ba6-42e7-4cb8-b04d-9f50939de2a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:51.010394 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.010363 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-kube-api-access-h6725" pod "d52a6ba6-42e7-4cb8-b04d-9f50939de2a5" (UID: "d52a6ba6-42e7-4cb8-b04d-9f50939de2a5"). InnerVolumeSpecName "kube-api-access-h6725". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:51.010527 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.010454 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-maas-api-xks-ca" pod "d52a6ba6-42e7-4cb8-b04d-9f50939de2a5" (UID: "d52a6ba6-42e7-4cb8-b04d-9f50939de2a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:51.010758 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.010723 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-kube-api-access-5hqks" pod "cfa2bd90-0708-4c95-aa5d-9df2b3d1965e" (UID: "cfa2bd90-0708-4c95-aa5d-9df2b3d1965e"). InnerVolumeSpecName "kube-api-access-5hqks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:51.011223 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.011204 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-maas-api-xks-ca" pod "cfa2bd90-0708-4c95-aa5d-9df2b3d1965e" (UID: "cfa2bd90-0708-4c95-aa5d-9df2b3d1965e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:51.109140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.108524 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:51.109140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.108553 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6725\" (UniqueName: \"kubernetes.io/projected/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-kube-api-access-h6725\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:51.109140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.108563 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:51.109140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.108573 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:51.109140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.108582 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:51.109140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.108590 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hqks\" (UniqueName: \"kubernetes.io/projected/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e-kube-api-access-5hqks\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:51.658618 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.658579 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz" event={"ID":"cfa2bd90-0708-4c95-aa5d-9df2b3d1965e","Type":"ContainerDied","Data":"f245bc7f5a5d122d0b735fa99730ebe8ee245e1b2f3b9dfff0f0595598139bab"} Nov 02 20:31:51.659006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.658629 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f245bc7f5a5d122d0b735fa99730ebe8ee245e1b2f3b9dfff0f0595598139bab" Nov 02 20:31:51.659006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.658690 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz" Nov 02 20:31:51.668513 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.666054 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" Nov 02 20:31:51.668513 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.666120 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb" event={"ID":"d52a6ba6-42e7-4cb8-b04d-9f50939de2a5","Type":"ContainerDied","Data":"e663027088ea634803bb536527301292449b892948c4fe7ef43a10e566d92b5e"} Nov 02 20:31:51.668513 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.666167 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e663027088ea634803bb536527301292449b892948c4fe7ef43a10e566d92b5e" Nov 02 20:31:51.669796 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.669775 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" event={"ID":"7baadcc3-0541-424e-9bc1-c28eb755787a","Type":"ContainerStarted","Data":"cfa8a89d1189486d45c644b4afcb0ae7ee4fde24dfc96a727183a1844d9ef7b7"} Nov 02 20:31:51.669853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.669815 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" event={"ID":"7baadcc3-0541-424e-9bc1-c28eb755787a","Type":"ContainerStarted","Data":"ba66043e1a83d48a99c28d1c834b3ee2ebe1c04bbb6dd63a35108bcb9822b46d"} Nov 02 20:31:51.806399 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.806349 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" podStartSLOduration=1.8063342900000001 podStartE2EDuration="1.80633429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:51.778700751 +0000 UTC m=+1040.068357299" watchObservedRunningTime="2026-11-02 20:31:51.80633429 +0000 UTC m=+1040.095990820" Nov 02 20:31:51.958489 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.958402 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:31:51.958489 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:51.958468 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" podUID="ebf56dbf-ece9-4da3-9589-77c972d8552e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 20:31:52.685108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:52.684262 4971 generic.go:356] "Generic (PLEG): container finished" podID="7baadcc3-0541-424e-9bc1-c28eb755787a" containerID="cfa8a89d1189486d45c644b4afcb0ae7ee4fde24dfc96a727183a1844d9ef7b7" exitCode=0 Nov 02 20:31:52.685656 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:52.685608 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" event={"ID":"7baadcc3-0541-424e-9bc1-c28eb755787a","Type":"ContainerDied","Data":"cfa8a89d1189486d45c644b4afcb0ae7ee4fde24dfc96a727183a1844d9ef7b7"} Nov 02 20:31:52.962163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:52.960746 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:31:52.962163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:52.960811 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" podUID="ebf56dbf-ece9-4da3-9589-77c972d8552e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 20:31:53.960685 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:53.959621 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:31:53.960685 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:53.959685 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" podUID="ebf56dbf-ece9-4da3-9589-77c972d8552e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 20:31:53.974288 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:53.973361 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:54.120858 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:54.120829 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" Nov 02 20:31:54.284158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:54.283324 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7baadcc3-0541-424e-9bc1-c28eb755787a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7baadcc3-0541-424e-9bc1-c28eb755787a-maas-api-openshift-ca\") pod \"7baadcc3-0541-424e-9bc1-c28eb755787a\" (UID: \"7baadcc3-0541-424e-9bc1-c28eb755787a\") " Nov 02 20:31:54.284158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:54.283366 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7baadcc3-0541-424e-9bc1-c28eb755787a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7baadcc3-0541-424e-9bc1-c28eb755787a-maas-api-xks-ca\") pod \"7baadcc3-0541-424e-9bc1-c28eb755787a\" (UID: \"7baadcc3-0541-424e-9bc1-c28eb755787a\") " Nov 02 20:31:54.284158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:54.283390 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7baadcc3-0541-424e-9bc1-c28eb755787a-kube-api-access-55kjj\" (UniqueName: \"kubernetes.io/projected/7baadcc3-0541-424e-9bc1-c28eb755787a-kube-api-access-55kjj\") pod \"7baadcc3-0541-424e-9bc1-c28eb755787a\" (UID: \"7baadcc3-0541-424e-9bc1-c28eb755787a\") " Nov 02 20:31:54.300107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:54.285330 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7baadcc3-0541-424e-9bc1-c28eb755787a-maas-api-openshift-ca" pod "7baadcc3-0541-424e-9bc1-c28eb755787a" (UID: "7baadcc3-0541-424e-9bc1-c28eb755787a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:54.300107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:54.288301 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7baadcc3-0541-424e-9bc1-c28eb755787a-kube-api-access-55kjj" pod "7baadcc3-0541-424e-9bc1-c28eb755787a" (UID: "7baadcc3-0541-424e-9bc1-c28eb755787a"). InnerVolumeSpecName "kube-api-access-55kjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:54.301111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:54.301044 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7baadcc3-0541-424e-9bc1-c28eb755787a-maas-api-xks-ca" pod "7baadcc3-0541-424e-9bc1-c28eb755787a" (UID: "7baadcc3-0541-424e-9bc1-c28eb755787a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:54.389006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:54.385258 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7baadcc3-0541-424e-9bc1-c28eb755787a-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:54.389006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:54.385293 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7baadcc3-0541-424e-9bc1-c28eb755787a-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:54.389006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:54.385309 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55kjj\" (UniqueName: \"kubernetes.io/projected/7baadcc3-0541-424e-9bc1-c28eb755787a-kube-api-access-55kjj\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:54.728146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:54.727988 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" event={"ID":"7baadcc3-0541-424e-9bc1-c28eb755787a","Type":"ContainerDied","Data":"ba66043e1a83d48a99c28d1c834b3ee2ebe1c04bbb6dd63a35108bcb9822b46d"} Nov 02 20:31:54.728146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:54.728056 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba66043e1a83d48a99c28d1c834b3ee2ebe1c04bbb6dd63a35108bcb9822b46d" Nov 02 20:31:54.728416 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:54.728163 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw" Nov 02 20:31:54.964115 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:54.960590 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:31:54.964115 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:54.960663 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" podUID="ebf56dbf-ece9-4da3-9589-77c972d8552e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 20:31:55.956623 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:55.956560 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:31:55.956623 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:55.956609 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" podUID="ebf56dbf-ece9-4da3-9589-77c972d8552e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 20:31:55.958607 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:55.958582 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2"] Nov 02 20:31:55.959126 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:55.959111 4971 cpu_manager.go:414] "containerMap: removing container" podUID="7baadcc3-0541-424e-9bc1-c28eb755787a" containerName="revoke-keys" Nov 02 20:31:55.959196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:55.959126 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7baadcc3-0541-424e-9bc1-c28eb755787a" containerName="revoke-keys" Nov 02 20:31:55.959196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:55.959147 4971 cpu_manager.go:414] "containerMap: removing container" podUID="cfa2bd90-0708-4c95-aa5d-9df2b3d1965e" containerName="revoke-keys" Nov 02 20:31:55.959196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:55.959153 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa2bd90-0708-4c95-aa5d-9df2b3d1965e" containerName="revoke-keys" Nov 02 20:31:55.959196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:55.959181 4971 cpu_manager.go:414] "containerMap: removing container" podUID="d52a6ba6-42e7-4cb8-b04d-9f50939de2a5" containerName="revoke-keys" Nov 02 20:31:55.959196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:55.959186 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52a6ba6-42e7-4cb8-b04d-9f50939de2a5" containerName="revoke-keys" Nov 02 20:31:55.959470 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:55.959270 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52a6ba6-42e7-4cb8-b04d-9f50939de2a5" containerName="revoke-keys" Nov 02 20:31:55.959470 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:55.959279 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="7baadcc3-0541-424e-9bc1-c28eb755787a" containerName="revoke-keys" Nov 02 20:31:55.959470 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:55.959289 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa2bd90-0708-4c95-aa5d-9df2b3d1965e" containerName="revoke-keys" Nov 02 20:31:55.964070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:55.964051 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2" Nov 02 20:31:55.982589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:55.982564 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2"] Nov 02 20:31:56.015887 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:56.015850 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:31:56.016454 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:56.016428 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7rs4\" (UniqueName: \"kubernetes.io/projected/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-kube-api-access-x7rs4\") pod \"maas-api-revoke-sub-c7d2aa81-c64f2\" (UID: \"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2" Nov 02 20:31:56.016578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:56.016465 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7d2aa81-c64f2\" (UID: \"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2" Nov 02 20:31:56.016578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:56.016483 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7d2aa81-c64f2\" (UID: \"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2" Nov 02 20:31:56.044110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:56.022199 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:31:56.117453 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:56.117424 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7rs4\" (UniqueName: \"kubernetes.io/projected/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-kube-api-access-x7rs4\") pod \"maas-api-revoke-sub-c7d2aa81-c64f2\" (UID: \"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2" Nov 02 20:31:56.117611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:56.117461 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7d2aa81-c64f2\" (UID: \"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2" Nov 02 20:31:56.117611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:56.117483 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7d2aa81-c64f2\" (UID: \"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2" Nov 02 20:31:56.118139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:56.118116 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7d2aa81-c64f2\" (UID: \"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2" Nov 02 20:31:56.120054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:56.120037 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7d2aa81-c64f2\" (UID: \"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2" Nov 02 20:31:56.175872 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:56.175840 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7rs4\" (UniqueName: \"kubernetes.io/projected/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-kube-api-access-x7rs4\") pod \"maas-api-revoke-sub-c7d2aa81-c64f2\" (UID: \"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2" Nov 02 20:31:56.283955 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:56.283884 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2" Nov 02 20:31:56.680988 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:56.680956 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2"] Nov 02 20:31:56.681623 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:31:56.681596 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e28c13_f0e4_4ebe_a8f7_192d1a184aae.slice/crio-823a156c88e1b27ec4a442ca5a16b9bdc64ac53bad12234e837c027377176848 WatchSource:0}: Error finding container 823a156c88e1b27ec4a442ca5a16b9bdc64ac53bad12234e837c027377176848: Status 404 returned error can't find the container with id 823a156c88e1b27ec4a442ca5a16b9bdc64ac53bad12234e837c027377176848 Nov 02 20:31:56.767185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:56.767147 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2" event={"ID":"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae","Type":"ContainerStarted","Data":"823a156c88e1b27ec4a442ca5a16b9bdc64ac53bad12234e837c027377176848"} Nov 02 20:31:56.956784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:56.956759 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:31:56.956895 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:56.956808 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" podUID="ebf56dbf-ece9-4da3-9589-77c972d8552e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 20:31:57.783271 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:57.783185 4971 generic.go:356] "Generic (PLEG): container finished" podID="d6e28c13-f0e4-4ebe-a8f7-192d1a184aae" containerID="1deefaaa9b841d87304dc627a41d9249a8450f7443ac1ac48f87d46d95a7496a" exitCode=0 Nov 02 20:31:57.783771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:57.783323 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2" event={"ID":"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae","Type":"ContainerDied","Data":"1deefaaa9b841d87304dc627a41d9249a8450f7443ac1ac48f87d46d95a7496a"} Nov 02 20:31:58.065434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:58.065338 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:58.066652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:58.066628 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:58.067531 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:58.067509 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:31:58.811578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:58.811547 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:31:58.942192 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:58.942160 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2" Nov 02 20:31:59.092319 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:59.092247 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-maas-api-openshift-ca\") pod \"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae\" (UID: \"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae\") " Nov 02 20:31:59.092482 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:59.092351 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-kube-api-access-x7rs4\" (UniqueName: \"kubernetes.io/projected/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-kube-api-access-x7rs4\") pod \"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae\" (UID: \"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae\") " Nov 02 20:31:59.092482 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:59.092393 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-maas-api-xks-ca\") pod \"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae\" (UID: \"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae\") " Nov 02 20:31:59.092642 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:59.092617 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-maas-api-openshift-ca" pod "d6e28c13-f0e4-4ebe-a8f7-192d1a184aae" (UID: "d6e28c13-f0e4-4ebe-a8f7-192d1a184aae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:59.092852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:59.092826 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:59.094631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:59.094607 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-maas-api-xks-ca" pod "d6e28c13-f0e4-4ebe-a8f7-192d1a184aae" (UID: "d6e28c13-f0e4-4ebe-a8f7-192d1a184aae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:59.094810 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:59.094788 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-kube-api-access-x7rs4" pod "d6e28c13-f0e4-4ebe-a8f7-192d1a184aae" (UID: "d6e28c13-f0e4-4ebe-a8f7-192d1a184aae"). InnerVolumeSpecName "kube-api-access-x7rs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:59.194294 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:59.194272 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7rs4\" (UniqueName: \"kubernetes.io/projected/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-kube-api-access-x7rs4\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:59.194294 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:59.194295 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:59.796879 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:59.796851 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2" Nov 02 20:31:59.797033 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:59.796889 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2" event={"ID":"d6e28c13-f0e4-4ebe-a8f7-192d1a184aae","Type":"ContainerDied","Data":"823a156c88e1b27ec4a442ca5a16b9bdc64ac53bad12234e837c027377176848"} Nov 02 20:31:59.797033 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:31:59.796936 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="823a156c88e1b27ec4a442ca5a16b9bdc64ac53bad12234e837c027377176848" Nov 02 20:32:02.566448 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.566404 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc"] Nov 02 20:32:02.567239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.567206 4971 cpu_manager.go:414] "containerMap: removing container" podUID="d6e28c13-f0e4-4ebe-a8f7-192d1a184aae" containerName="revoke-keys" Nov 02 20:32:02.567239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.567238 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e28c13-f0e4-4ebe-a8f7-192d1a184aae" containerName="revoke-keys" Nov 02 20:32:02.567455 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.567441 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e28c13-f0e4-4ebe-a8f7-192d1a184aae" containerName="revoke-keys" Nov 02 20:32:02.571175 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.571145 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc" Nov 02 20:32:02.624265 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.624239 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc"] Nov 02 20:32:02.667264 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.667203 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:32:02.667407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.667284 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:32:02.736953 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.736925 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdcrw\" (UniqueName: \"kubernetes.io/projected/f72d94b0-291c-421a-90c1-30b48b5ac842-kube-api-access-qdcrw\") pod \"maas-api-revoke-sub-6f4518cf-qv7lc\" (UID: \"f72d94b0-291c-421a-90c1-30b48b5ac842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc" Nov 02 20:32:02.737077 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.736966 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72d94b0-291c-421a-90c1-30b48b5ac842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f4518cf-qv7lc\" (UID: \"f72d94b0-291c-421a-90c1-30b48b5ac842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc" Nov 02 20:32:02.737077 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.737057 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72d94b0-291c-421a-90c1-30b48b5ac842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f4518cf-qv7lc\" (UID: \"f72d94b0-291c-421a-90c1-30b48b5ac842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc" Nov 02 20:32:02.838152 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.838062 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdcrw\" (UniqueName: \"kubernetes.io/projected/f72d94b0-291c-421a-90c1-30b48b5ac842-kube-api-access-qdcrw\") pod \"maas-api-revoke-sub-6f4518cf-qv7lc\" (UID: \"f72d94b0-291c-421a-90c1-30b48b5ac842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc" Nov 02 20:32:02.838281 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.838152 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72d94b0-291c-421a-90c1-30b48b5ac842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f4518cf-qv7lc\" (UID: \"f72d94b0-291c-421a-90c1-30b48b5ac842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc" Nov 02 20:32:02.838281 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.838246 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72d94b0-291c-421a-90c1-30b48b5ac842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f4518cf-qv7lc\" (UID: \"f72d94b0-291c-421a-90c1-30b48b5ac842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc" Nov 02 20:32:02.838938 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.838912 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72d94b0-291c-421a-90c1-30b48b5ac842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f4518cf-qv7lc\" (UID: \"f72d94b0-291c-421a-90c1-30b48b5ac842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc" Nov 02 20:32:02.840721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.840693 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72d94b0-291c-421a-90c1-30b48b5ac842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f4518cf-qv7lc\" (UID: \"f72d94b0-291c-421a-90c1-30b48b5ac842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc" Nov 02 20:32:02.896098 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:02.896065 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdcrw\" (UniqueName: \"kubernetes.io/projected/f72d94b0-291c-421a-90c1-30b48b5ac842-kube-api-access-qdcrw\") pod \"maas-api-revoke-sub-6f4518cf-qv7lc\" (UID: \"f72d94b0-291c-421a-90c1-30b48b5ac842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc" Nov 02 20:32:03.010203 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.010170 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2"] Nov 02 20:32:03.014689 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.014672 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2" Nov 02 20:32:03.068907 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.068884 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2"] Nov 02 20:32:03.141700 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.141669 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9sbd\" (UniqueName: \"kubernetes.io/projected/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-kube-api-access-w9sbd\") pod \"maas-api-revoke-sub-446ad717-l2xc2\" (UID: \"da0ae96b-7041-4e4b-bc18-84d76a5d9e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2" Nov 02 20:32:03.141862 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.141841 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-446ad717-l2xc2\" (UID: \"da0ae96b-7041-4e4b-bc18-84d76a5d9e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2" Nov 02 20:32:03.141926 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.141916 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-446ad717-l2xc2\" (UID: \"da0ae96b-7041-4e4b-bc18-84d76a5d9e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2" Nov 02 20:32:03.144574 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.144553 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb"] Nov 02 20:32:03.148521 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.148505 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb" Nov 02 20:32:03.185831 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.185809 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc" Nov 02 20:32:03.205229 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.205204 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb"] Nov 02 20:32:03.243413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.243378 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9sbd\" (UniqueName: \"kubernetes.io/projected/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-kube-api-access-w9sbd\") pod \"maas-api-revoke-sub-446ad717-l2xc2\" (UID: \"da0ae96b-7041-4e4b-bc18-84d76a5d9e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2" Nov 02 20:32:03.243719 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.243688 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4459c350-cd02-4b0e-b28f-98e0ead6a077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c9cffb8-4b5jb\" (UID: \"4459c350-cd02-4b0e-b28f-98e0ead6a077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb" Nov 02 20:32:03.243865 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.243797 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttgrx\" (UniqueName: \"kubernetes.io/projected/4459c350-cd02-4b0e-b28f-98e0ead6a077-kube-api-access-ttgrx\") pod \"maas-api-revoke-sub-7c9cffb8-4b5jb\" (UID: \"4459c350-cd02-4b0e-b28f-98e0ead6a077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb" Nov 02 20:32:03.243865 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.243820 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-446ad717-l2xc2\" (UID: \"da0ae96b-7041-4e4b-bc18-84d76a5d9e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2" Nov 02 20:32:03.244015 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.243876 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4459c350-cd02-4b0e-b28f-98e0ead6a077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c9cffb8-4b5jb\" (UID: \"4459c350-cd02-4b0e-b28f-98e0ead6a077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb" Nov 02 20:32:03.244015 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.243898 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-446ad717-l2xc2\" (UID: \"da0ae96b-7041-4e4b-bc18-84d76a5d9e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2" Nov 02 20:32:03.244870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.244846 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-446ad717-l2xc2\" (UID: \"da0ae96b-7041-4e4b-bc18-84d76a5d9e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2" Nov 02 20:32:03.246701 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.246677 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-446ad717-l2xc2\" (UID: \"da0ae96b-7041-4e4b-bc18-84d76a5d9e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2" Nov 02 20:32:03.311801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.311764 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9sbd\" (UniqueName: \"kubernetes.io/projected/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-kube-api-access-w9sbd\") pod \"maas-api-revoke-sub-446ad717-l2xc2\" (UID: \"da0ae96b-7041-4e4b-bc18-84d76a5d9e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2" Nov 02 20:32:03.329484 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.329262 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2" Nov 02 20:32:03.345366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.345259 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4459c350-cd02-4b0e-b28f-98e0ead6a077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c9cffb8-4b5jb\" (UID: \"4459c350-cd02-4b0e-b28f-98e0ead6a077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb" Nov 02 20:32:03.345366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.345321 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgrx\" (UniqueName: \"kubernetes.io/projected/4459c350-cd02-4b0e-b28f-98e0ead6a077-kube-api-access-ttgrx\") pod \"maas-api-revoke-sub-7c9cffb8-4b5jb\" (UID: \"4459c350-cd02-4b0e-b28f-98e0ead6a077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb" Nov 02 20:32:03.345366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.345365 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4459c350-cd02-4b0e-b28f-98e0ead6a077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c9cffb8-4b5jb\" (UID: \"4459c350-cd02-4b0e-b28f-98e0ead6a077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb" Nov 02 20:32:03.346965 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.346656 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4459c350-cd02-4b0e-b28f-98e0ead6a077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c9cffb8-4b5jb\" (UID: \"4459c350-cd02-4b0e-b28f-98e0ead6a077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb" Nov 02 20:32:03.348883 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:32:03.348850 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:32:03.356304 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.356279 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4459c350-cd02-4b0e-b28f-98e0ead6a077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c9cffb8-4b5jb\" (UID: \"4459c350-cd02-4b0e-b28f-98e0ead6a077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb" Nov 02 20:32:03.404319 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.404298 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgrx\" (UniqueName: \"kubernetes.io/projected/4459c350-cd02-4b0e-b28f-98e0ead6a077-kube-api-access-ttgrx\") pod \"maas-api-revoke-sub-7c9cffb8-4b5jb\" (UID: \"4459c350-cd02-4b0e-b28f-98e0ead6a077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb" Nov 02 20:32:03.461815 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.461757 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb" Nov 02 20:32:03.577965 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.577937 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc"] Nov 02 20:32:03.578958 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:32:03.578929 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72d94b0_291c_421a_90c1_30b48b5ac842.slice/crio-a6485dbc0f010b9a62af0431a18f578a144ba37777d5472bb55b279fd8c59e9e WatchSource:0}: Error finding container a6485dbc0f010b9a62af0431a18f578a144ba37777d5472bb55b279fd8c59e9e: Status 404 returned error can't find the container with id a6485dbc0f010b9a62af0431a18f578a144ba37777d5472bb55b279fd8c59e9e Nov 02 20:32:03.706446 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:32:03.706415 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0ae96b_7041_4e4b_bc18_84d76a5d9e51.slice/crio-337967a5cc07375fc254a8bb2b017afe8864ae021294a5eb34171a70c4583b93 WatchSource:0}: Error finding container 337967a5cc07375fc254a8bb2b017afe8864ae021294a5eb34171a70c4583b93: Status 404 returned error can't find the container with id 337967a5cc07375fc254a8bb2b017afe8864ae021294a5eb34171a70c4583b93 Nov 02 20:32:03.707430 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.707406 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2"] Nov 02 20:32:03.721769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.721650 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:03.827268 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.827240 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc" event={"ID":"f72d94b0-291c-421a-90c1-30b48b5ac842","Type":"ContainerStarted","Data":"a6485dbc0f010b9a62af0431a18f578a144ba37777d5472bb55b279fd8c59e9e"} Nov 02 20:32:03.828644 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.828620 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2" event={"ID":"da0ae96b-7041-4e4b-bc18-84d76a5d9e51","Type":"ContainerStarted","Data":"337967a5cc07375fc254a8bb2b017afe8864ae021294a5eb34171a70c4583b93"} Nov 02 20:32:03.836246 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:03.836221 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb"] Nov 02 20:32:03.924832 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:32:03.924793 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4459c350_cd02_4b0e_b28f_98e0ead6a077.slice/crio-a6ba144592b2e624ba92f8af32c386f495d6d7fb334900f7a124295ee9440060 WatchSource:0}: Error finding container a6ba144592b2e624ba92f8af32c386f495d6d7fb334900f7a124295ee9440060: Status 404 returned error can't find the container with id a6ba144592b2e624ba92f8af32c386f495d6d7fb334900f7a124295ee9440060 Nov 02 20:32:04.836776 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:04.836740 4971 generic.go:356] "Generic (PLEG): container finished" podID="f72d94b0-291c-421a-90c1-30b48b5ac842" containerID="c49817ec72189b8d1a44ed95baecf4a57a8082c59125c41361ab5a28a2cd8de9" exitCode=0 Nov 02 20:32:04.837134 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:04.836826 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc" event={"ID":"f72d94b0-291c-421a-90c1-30b48b5ac842","Type":"ContainerDied","Data":"c49817ec72189b8d1a44ed95baecf4a57a8082c59125c41361ab5a28a2cd8de9"} Nov 02 20:32:04.838451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:04.838432 4971 generic.go:356] "Generic (PLEG): container finished" podID="4459c350-cd02-4b0e-b28f-98e0ead6a077" containerID="ebbe18bdd7db93087666f82f1d0029349f6e5ab70cd262fa52cf25553b690850" exitCode=0 Nov 02 20:32:04.838538 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:04.838512 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb" event={"ID":"4459c350-cd02-4b0e-b28f-98e0ead6a077","Type":"ContainerDied","Data":"ebbe18bdd7db93087666f82f1d0029349f6e5ab70cd262fa52cf25553b690850"} Nov 02 20:32:04.838575 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:04.838553 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb" event={"ID":"4459c350-cd02-4b0e-b28f-98e0ead6a077","Type":"ContainerStarted","Data":"a6ba144592b2e624ba92f8af32c386f495d6d7fb334900f7a124295ee9440060"} Nov 02 20:32:04.840046 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:04.840028 4971 generic.go:356] "Generic (PLEG): container finished" podID="da0ae96b-7041-4e4b-bc18-84d76a5d9e51" containerID="5f46c0a8a975accbadefd043f2000b192179c4aabf1b1290412970f5ccb7aa5e" exitCode=0 Nov 02 20:32:04.840132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:04.840112 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2" event={"ID":"da0ae96b-7041-4e4b-bc18-84d76a5d9e51","Type":"ContainerDied","Data":"5f46c0a8a975accbadefd043f2000b192179c4aabf1b1290412970f5ccb7aa5e"} Nov 02 20:32:05.332056 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:05.332026 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm"] Nov 02 20:32:05.336198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:05.336179 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" Nov 02 20:32:05.390200 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:05.390167 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm"] Nov 02 20:32:05.468641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:05.468613 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39568bf0-9145-4591-856a-80feed1768aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa722c67-nkqzm\" (UID: \"39568bf0-9145-4591-856a-80feed1768aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" Nov 02 20:32:05.468791 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:05.468663 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc8pf\" (UniqueName: \"kubernetes.io/projected/39568bf0-9145-4591-856a-80feed1768aa-kube-api-access-qc8pf\") pod \"maas-api-revoke-sub-fa722c67-nkqzm\" (UID: \"39568bf0-9145-4591-856a-80feed1768aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" Nov 02 20:32:05.468878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:05.468814 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39568bf0-9145-4591-856a-80feed1768aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa722c67-nkqzm\" (UID: \"39568bf0-9145-4591-856a-80feed1768aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" Nov 02 20:32:05.569502 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:05.569474 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39568bf0-9145-4591-856a-80feed1768aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa722c67-nkqzm\" (UID: \"39568bf0-9145-4591-856a-80feed1768aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" Nov 02 20:32:05.569641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:05.569524 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc8pf\" (UniqueName: \"kubernetes.io/projected/39568bf0-9145-4591-856a-80feed1768aa-kube-api-access-qc8pf\") pod \"maas-api-revoke-sub-fa722c67-nkqzm\" (UID: \"39568bf0-9145-4591-856a-80feed1768aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" Nov 02 20:32:05.569641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:05.569600 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39568bf0-9145-4591-856a-80feed1768aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa722c67-nkqzm\" (UID: \"39568bf0-9145-4591-856a-80feed1768aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" Nov 02 20:32:05.570227 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:05.570203 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39568bf0-9145-4591-856a-80feed1768aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa722c67-nkqzm\" (UID: \"39568bf0-9145-4591-856a-80feed1768aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" Nov 02 20:32:05.571805 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:05.571784 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39568bf0-9145-4591-856a-80feed1768aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa722c67-nkqzm\" (UID: \"39568bf0-9145-4591-856a-80feed1768aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" Nov 02 20:32:05.631154 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:05.631130 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc8pf\" (UniqueName: \"kubernetes.io/projected/39568bf0-9145-4591-856a-80feed1768aa-kube-api-access-qc8pf\") pod \"maas-api-revoke-sub-fa722c67-nkqzm\" (UID: \"39568bf0-9145-4591-856a-80feed1768aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" Nov 02 20:32:05.649939 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:05.649918 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" Nov 02 20:32:06.015503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.015474 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb" Nov 02 20:32:06.033285 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:32:06.033193 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39568bf0_9145_4591_856a_80feed1768aa.slice/crio-84a0367b5081fea86e4d562876579d2203819e7fed0b1c977a1f5715022b8513 WatchSource:0}: Error finding container 84a0367b5081fea86e4d562876579d2203819e7fed0b1c977a1f5715022b8513: Status 404 returned error can't find the container with id 84a0367b5081fea86e4d562876579d2203819e7fed0b1c977a1f5715022b8513 Nov 02 20:32:06.034002 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.033978 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm"] Nov 02 20:32:06.058430 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.058348 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc" Nov 02 20:32:06.095356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.095337 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2" Nov 02 20:32:06.175823 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.175743 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72d94b0-291c-421a-90c1-30b48b5ac842-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72d94b0-291c-421a-90c1-30b48b5ac842-maas-api-openshift-ca\") pod \"f72d94b0-291c-421a-90c1-30b48b5ac842\" (UID: \"f72d94b0-291c-421a-90c1-30b48b5ac842\") " Nov 02 20:32:06.175984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.175843 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f72d94b0-291c-421a-90c1-30b48b5ac842-kube-api-access-qdcrw\" (UniqueName: \"kubernetes.io/projected/f72d94b0-291c-421a-90c1-30b48b5ac842-kube-api-access-qdcrw\") pod \"f72d94b0-291c-421a-90c1-30b48b5ac842\" (UID: \"f72d94b0-291c-421a-90c1-30b48b5ac842\") " Nov 02 20:32:06.175984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.175869 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4459c350-cd02-4b0e-b28f-98e0ead6a077-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4459c350-cd02-4b0e-b28f-98e0ead6a077-maas-api-openshift-ca\") pod \"4459c350-cd02-4b0e-b28f-98e0ead6a077\" (UID: \"4459c350-cd02-4b0e-b28f-98e0ead6a077\") " Nov 02 20:32:06.175984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.175905 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f72d94b0-291c-421a-90c1-30b48b5ac842-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72d94b0-291c-421a-90c1-30b48b5ac842-maas-api-xks-ca\") pod \"f72d94b0-291c-421a-90c1-30b48b5ac842\" (UID: \"f72d94b0-291c-421a-90c1-30b48b5ac842\") " Nov 02 20:32:06.175984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.175973 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-maas-api-openshift-ca\") pod \"da0ae96b-7041-4e4b-bc18-84d76a5d9e51\" (UID: \"da0ae96b-7041-4e4b-bc18-84d76a5d9e51\") " Nov 02 20:32:06.176303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.176023 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-kube-api-access-w9sbd\" (UniqueName: \"kubernetes.io/projected/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-kube-api-access-w9sbd\") pod \"da0ae96b-7041-4e4b-bc18-84d76a5d9e51\" (UID: \"da0ae96b-7041-4e4b-bc18-84d76a5d9e51\") " Nov 02 20:32:06.176303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.176074 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4459c350-cd02-4b0e-b28f-98e0ead6a077-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4459c350-cd02-4b0e-b28f-98e0ead6a077-maas-api-xks-ca\") pod \"4459c350-cd02-4b0e-b28f-98e0ead6a077\" (UID: \"4459c350-cd02-4b0e-b28f-98e0ead6a077\") " Nov 02 20:32:06.176303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.176167 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4459c350-cd02-4b0e-b28f-98e0ead6a077-kube-api-access-ttgrx\" (UniqueName: \"kubernetes.io/projected/4459c350-cd02-4b0e-b28f-98e0ead6a077-kube-api-access-ttgrx\") pod \"4459c350-cd02-4b0e-b28f-98e0ead6a077\" (UID: \"4459c350-cd02-4b0e-b28f-98e0ead6a077\") " Nov 02 20:32:06.176303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.176190 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72d94b0-291c-421a-90c1-30b48b5ac842-maas-api-openshift-ca" pod "f72d94b0-291c-421a-90c1-30b48b5ac842" (UID: "f72d94b0-291c-421a-90c1-30b48b5ac842"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:06.176303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.176273 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-maas-api-xks-ca\") pod \"da0ae96b-7041-4e4b-bc18-84d76a5d9e51\" (UID: \"da0ae96b-7041-4e4b-bc18-84d76a5d9e51\") " Nov 02 20:32:06.176568 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.176304 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4459c350-cd02-4b0e-b28f-98e0ead6a077-maas-api-openshift-ca" pod "4459c350-cd02-4b0e-b28f-98e0ead6a077" (UID: "4459c350-cd02-4b0e-b28f-98e0ead6a077"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:06.176568 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.176413 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-maas-api-openshift-ca" pod "da0ae96b-7041-4e4b-bc18-84d76a5d9e51" (UID: "da0ae96b-7041-4e4b-bc18-84d76a5d9e51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:06.176973 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.176949 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:06.177114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.176977 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72d94b0-291c-421a-90c1-30b48b5ac842-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:06.177114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.176994 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4459c350-cd02-4b0e-b28f-98e0ead6a077-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:06.178340 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.178308 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72d94b0-291c-421a-90c1-30b48b5ac842-kube-api-access-qdcrw" pod "f72d94b0-291c-421a-90c1-30b48b5ac842" (UID: "f72d94b0-291c-421a-90c1-30b48b5ac842"). InnerVolumeSpecName "kube-api-access-qdcrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:06.178420 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.178392 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-kube-api-access-w9sbd" pod "da0ae96b-7041-4e4b-bc18-84d76a5d9e51" (UID: "da0ae96b-7041-4e4b-bc18-84d76a5d9e51"). InnerVolumeSpecName "kube-api-access-w9sbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:06.178776 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.178760 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72d94b0-291c-421a-90c1-30b48b5ac842-maas-api-xks-ca" pod "f72d94b0-291c-421a-90c1-30b48b5ac842" (UID: "f72d94b0-291c-421a-90c1-30b48b5ac842"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:06.179008 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.178989 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4459c350-cd02-4b0e-b28f-98e0ead6a077-kube-api-access-ttgrx" pod "4459c350-cd02-4b0e-b28f-98e0ead6a077" (UID: "4459c350-cd02-4b0e-b28f-98e0ead6a077"). InnerVolumeSpecName "kube-api-access-ttgrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:06.179008 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.178995 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-maas-api-xks-ca" pod "da0ae96b-7041-4e4b-bc18-84d76a5d9e51" (UID: "da0ae96b-7041-4e4b-bc18-84d76a5d9e51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:06.179160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.178999 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4459c350-cd02-4b0e-b28f-98e0ead6a077-maas-api-xks-ca" pod "4459c350-cd02-4b0e-b28f-98e0ead6a077" (UID: "4459c350-cd02-4b0e-b28f-98e0ead6a077"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:06.278524 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.278499 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdcrw\" (UniqueName: \"kubernetes.io/projected/f72d94b0-291c-421a-90c1-30b48b5ac842-kube-api-access-qdcrw\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:06.278524 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.278525 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72d94b0-291c-421a-90c1-30b48b5ac842-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:06.278681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.278535 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9sbd\" (UniqueName: \"kubernetes.io/projected/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-kube-api-access-w9sbd\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:06.278681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.278543 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4459c350-cd02-4b0e-b28f-98e0ead6a077-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:06.278681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.278553 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttgrx\" (UniqueName: \"kubernetes.io/projected/4459c350-cd02-4b0e-b28f-98e0ead6a077-kube-api-access-ttgrx\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:06.278681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.278561 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0ae96b-7041-4e4b-bc18-84d76a5d9e51-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:06.855119 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.855059 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" event={"ID":"39568bf0-9145-4591-856a-80feed1768aa","Type":"ContainerStarted","Data":"a76c647af93e9d072405ebf61e41e667661f4225dc18035c6349e79db58f41a2"} Nov 02 20:32:06.855271 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.855141 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" event={"ID":"39568bf0-9145-4591-856a-80feed1768aa","Type":"ContainerStarted","Data":"84a0367b5081fea86e4d562876579d2203819e7fed0b1c977a1f5715022b8513"} Nov 02 20:32:06.856441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.856419 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc" Nov 02 20:32:06.856566 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.856456 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc" event={"ID":"f72d94b0-291c-421a-90c1-30b48b5ac842","Type":"ContainerDied","Data":"a6485dbc0f010b9a62af0431a18f578a144ba37777d5472bb55b279fd8c59e9e"} Nov 02 20:32:06.856566 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.856501 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6485dbc0f010b9a62af0431a18f578a144ba37777d5472bb55b279fd8c59e9e" Nov 02 20:32:06.857771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.857754 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb" Nov 02 20:32:06.857874 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.857803 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb" event={"ID":"4459c350-cd02-4b0e-b28f-98e0ead6a077","Type":"ContainerDied","Data":"a6ba144592b2e624ba92f8af32c386f495d6d7fb334900f7a124295ee9440060"} Nov 02 20:32:06.857874 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.857845 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6ba144592b2e624ba92f8af32c386f495d6d7fb334900f7a124295ee9440060" Nov 02 20:32:06.859317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.859295 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2" event={"ID":"da0ae96b-7041-4e4b-bc18-84d76a5d9e51","Type":"ContainerDied","Data":"337967a5cc07375fc254a8bb2b017afe8864ae021294a5eb34171a70c4583b93"} Nov 02 20:32:06.859317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.859312 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2" Nov 02 20:32:06.859483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.859329 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="337967a5cc07375fc254a8bb2b017afe8864ae021294a5eb34171a70c4583b93" Nov 02 20:32:06.965057 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:06.965012 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" podStartSLOduration=1.964999375 podStartE2EDuration="1.964999375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:06.963493889 +0000 UTC m=+1055.253150430" watchObservedRunningTime="2026-11-02 20:32:06.964999375 +0000 UTC m=+1055.254655928" Nov 02 20:32:07.868440 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:07.868410 4971 generic.go:356] "Generic (PLEG): container finished" podID="39568bf0-9145-4591-856a-80feed1768aa" containerID="a76c647af93e9d072405ebf61e41e667661f4225dc18035c6349e79db58f41a2" exitCode=0 Nov 02 20:32:07.869144 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:07.869121 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" event={"ID":"39568bf0-9145-4591-856a-80feed1768aa","Type":"ContainerDied","Data":"a76c647af93e9d072405ebf61e41e667661f4225dc18035c6349e79db58f41a2"} Nov 02 20:32:08.364115 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.364070 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp"] Nov 02 20:32:08.364671 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.364656 4971 cpu_manager.go:414] "containerMap: removing container" podUID="f72d94b0-291c-421a-90c1-30b48b5ac842" containerName="revoke-keys" Nov 02 20:32:08.364718 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.364674 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f72d94b0-291c-421a-90c1-30b48b5ac842" containerName="revoke-keys" Nov 02 20:32:08.364718 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.364691 4971 cpu_manager.go:414] "containerMap: removing container" podUID="da0ae96b-7041-4e4b-bc18-84d76a5d9e51" containerName="revoke-keys" Nov 02 20:32:08.364718 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.364696 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0ae96b-7041-4e4b-bc18-84d76a5d9e51" containerName="revoke-keys" Nov 02 20:32:08.364718 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.364709 4971 cpu_manager.go:414] "containerMap: removing container" podUID="4459c350-cd02-4b0e-b28f-98e0ead6a077" containerName="revoke-keys" Nov 02 20:32:08.364718 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.364714 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4459c350-cd02-4b0e-b28f-98e0ead6a077" containerName="revoke-keys" Nov 02 20:32:08.364866 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.364795 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="f72d94b0-291c-421a-90c1-30b48b5ac842" containerName="revoke-keys" Nov 02 20:32:08.364866 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.364807 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0ae96b-7041-4e4b-bc18-84d76a5d9e51" containerName="revoke-keys" Nov 02 20:32:08.364866 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.364814 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="4459c350-cd02-4b0e-b28f-98e0ead6a077" containerName="revoke-keys" Nov 02 20:32:08.368329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.368313 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.447808 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.447777 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp"] Nov 02 20:32:08.490194 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.490152 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:08.502608 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.502581 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.502782 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.502620 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.502782 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.502643 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.502902 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.502793 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.502953 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.502894 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.502999 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.502957 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.503039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.503006 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.503104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.503039 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.503104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.503058 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsr9n\" (UniqueName: \"kubernetes.io/projected/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-kube-api-access-xsr9n\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.604027 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.603987 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.604027 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.604023 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.604274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.604064 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.604274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.604160 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.604274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.604209 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.604274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.604255 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.604487 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.604291 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.604487 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.604318 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsr9n\" (UniqueName: \"kubernetes.io/projected/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-kube-api-access-xsr9n\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.604487 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.604421 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.604487 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.604458 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.604699 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.604520 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.604749 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.604725 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.604946 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.604921 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.605029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.604999 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.606976 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.606956 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.607135 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.607117 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.664918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.664855 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.665844 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.665815 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsr9n\" (UniqueName: \"kubernetes.io/projected/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-kube-api-access-xsr9n\") pod \"maas-default-gateway-openshift-default-85544d4cd7-g69pp\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:08.682159 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:08.682142 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:09.015712 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.015685 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" Nov 02 20:32:09.103938 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:32:09.103907 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a8fe63_b184_4aa9_816b_eeb3c161ebbe.slice/crio-7db14d4772df130ae7b32fcfbf6def1c0603feb1cd90446191f2563aab9781e6 WatchSource:0}: Error finding container 7db14d4772df130ae7b32fcfbf6def1c0603feb1cd90446191f2563aab9781e6: Status 404 returned error can't find the container with id 7db14d4772df130ae7b32fcfbf6def1c0603feb1cd90446191f2563aab9781e6 Nov 02 20:32:09.104323 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.104298 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp"] Nov 02 20:32:09.106682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.106650 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:09.106759 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.106714 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:09.106759 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.106743 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:09.109025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.109007 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39568bf0-9145-4591-856a-80feed1768aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39568bf0-9145-4591-856a-80feed1768aa-maas-api-openshift-ca\") pod \"39568bf0-9145-4591-856a-80feed1768aa\" (UID: \"39568bf0-9145-4591-856a-80feed1768aa\") " Nov 02 20:32:09.109178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.109164 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39568bf0-9145-4591-856a-80feed1768aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39568bf0-9145-4591-856a-80feed1768aa-maas-api-xks-ca\") pod \"39568bf0-9145-4591-856a-80feed1768aa\" (UID: \"39568bf0-9145-4591-856a-80feed1768aa\") " Nov 02 20:32:09.109242 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.109227 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39568bf0-9145-4591-856a-80feed1768aa-kube-api-access-qc8pf\" (UniqueName: \"kubernetes.io/projected/39568bf0-9145-4591-856a-80feed1768aa-kube-api-access-qc8pf\") pod \"39568bf0-9145-4591-856a-80feed1768aa\" (UID: \"39568bf0-9145-4591-856a-80feed1768aa\") " Nov 02 20:32:09.109483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.109387 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39568bf0-9145-4591-856a-80feed1768aa-maas-api-openshift-ca" pod "39568bf0-9145-4591-856a-80feed1768aa" (UID: "39568bf0-9145-4591-856a-80feed1768aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:09.109638 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.109622 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39568bf0-9145-4591-856a-80feed1768aa-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:09.111745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.111724 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39568bf0-9145-4591-856a-80feed1768aa-maas-api-xks-ca" pod "39568bf0-9145-4591-856a-80feed1768aa" (UID: "39568bf0-9145-4591-856a-80feed1768aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:09.111745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.111732 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39568bf0-9145-4591-856a-80feed1768aa-kube-api-access-qc8pf" pod "39568bf0-9145-4591-856a-80feed1768aa" (UID: "39568bf0-9145-4591-856a-80feed1768aa"). InnerVolumeSpecName "kube-api-access-qc8pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:09.211153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.211073 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39568bf0-9145-4591-856a-80feed1768aa-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:09.211153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.211124 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc8pf\" (UniqueName: \"kubernetes.io/projected/39568bf0-9145-4591-856a-80feed1768aa-kube-api-access-qc8pf\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:09.884541 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.884510 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" event={"ID":"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe","Type":"ContainerStarted","Data":"cf5042cffb0b6ec65d73a4cb8b8697be704113ffebdd1b558d161b99ad6d460c"} Nov 02 20:32:09.884672 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.884563 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" event={"ID":"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe","Type":"ContainerStarted","Data":"7db14d4772df130ae7b32fcfbf6def1c0603feb1cd90446191f2563aab9781e6"} Nov 02 20:32:09.885910 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.885888 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" Nov 02 20:32:09.886028 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.885916 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm" event={"ID":"39568bf0-9145-4591-856a-80feed1768aa","Type":"ContainerDied","Data":"84a0367b5081fea86e4d562876579d2203819e7fed0b1c977a1f5715022b8513"} Nov 02 20:32:09.886028 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:09.885954 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84a0367b5081fea86e4d562876579d2203819e7fed0b1c977a1f5715022b8513" Nov 02 20:32:10.048891 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:10.048826 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" podStartSLOduration=2.048806578 podStartE2EDuration="2.048806578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:10.011848482 +0000 UTC m=+1058.301505010" watchObservedRunningTime="2026-11-02 20:32:10.048806578 +0000 UTC m=+1058.338463108" Nov 02 20:32:10.683930 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:10.682510 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:10.684099 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:10.683940 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-g69pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:32:10.684099 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:10.684001 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" podUID="b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 20:32:11.699148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:11.694197 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-g69pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:32:11.699148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:11.694274 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" podUID="b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 20:32:12.605288 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.605259 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw"] Nov 02 20:32:12.605827 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.605813 4971 cpu_manager.go:414] "containerMap: removing container" podUID="39568bf0-9145-4591-856a-80feed1768aa" containerName="revoke-keys" Nov 02 20:32:12.605827 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.605828 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39568bf0-9145-4591-856a-80feed1768aa" containerName="revoke-keys" Nov 02 20:32:12.605961 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.605949 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="39568bf0-9145-4591-856a-80feed1768aa" containerName="revoke-keys" Nov 02 20:32:12.609661 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.609645 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.668478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.668445 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw"] Nov 02 20:32:12.683413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.683385 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-g69pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:32:12.683559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.683435 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" podUID="b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 20:32:12.708508 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.708485 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bce768\"/\"distinct-w4-bce768-kserve-self-signed-certs\"" Nov 02 20:32:12.789837 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.789801 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10dd59b7-c045-4b1a-a27e-b7034e0b653d-tls-certs\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.790013 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.789871 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtzbp\" (UniqueName: \"kubernetes.io/projected/10dd59b7-c045-4b1a-a27e-b7034e0b653d-kube-api-access-rtzbp\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.790013 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.789959 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-home\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.790013 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.789990 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-model-cache\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.790140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.790127 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-dshm\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.790193 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.790175 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-tmp-dir\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.863034 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.862962 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:12.891530 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.891505 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-tmp-dir\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.891670 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.891586 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10dd59b7-c045-4b1a-a27e-b7034e0b653d-tls-certs\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.891670 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.891634 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtzbp\" (UniqueName: \"kubernetes.io/projected/10dd59b7-c045-4b1a-a27e-b7034e0b653d-kube-api-access-rtzbp\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.891670 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.891653 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-home\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.891798 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.891776 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-model-cache\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.891897 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.891880 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-dshm\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.892014 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.891905 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-tmp-dir\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.892014 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.891969 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-home\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.892162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.892142 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-model-cache\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.894037 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.894019 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-dshm\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.894145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.894020 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10dd59b7-c045-4b1a-a27e-b7034e0b653d-tls-certs\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:12.958023 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:12.957958 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtzbp\" (UniqueName: \"kubernetes.io/projected/10dd59b7-c045-4b1a-a27e-b7034e0b653d-kube-api-access-rtzbp\") pod \"distinct-w4-bce768-kserve-c5795f59b-2vdnw\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:13.241356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.241265 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:13.381285 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.381235 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq"] Nov 02 20:32:13.397577 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.396470 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" Nov 02 20:32:13.422113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.419671 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq"] Nov 02 20:32:13.453109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.449570 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:32:13.453109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.450260 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:32:13.458126 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.456146 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:13.499680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.499608 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fe9460-fed6-4e8d-8625-5915605902e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46b8e40d-h7bqq\" (UID: \"32fe9460-fed6-4e8d-8625-5915605902e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" Nov 02 20:32:13.499680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.499652 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fe9460-fed6-4e8d-8625-5915605902e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46b8e40d-h7bqq\" (UID: \"32fe9460-fed6-4e8d-8625-5915605902e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" Nov 02 20:32:13.499877 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.499826 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pq4d\" (UniqueName: \"kubernetes.io/projected/32fe9460-fed6-4e8d-8625-5915605902e0-kube-api-access-2pq4d\") pod \"maas-api-revoke-sub-46b8e40d-h7bqq\" (UID: \"32fe9460-fed6-4e8d-8625-5915605902e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" Nov 02 20:32:13.600529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.600478 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pq4d\" (UniqueName: \"kubernetes.io/projected/32fe9460-fed6-4e8d-8625-5915605902e0-kube-api-access-2pq4d\") pod \"maas-api-revoke-sub-46b8e40d-h7bqq\" (UID: \"32fe9460-fed6-4e8d-8625-5915605902e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" Nov 02 20:32:13.600678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.600590 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fe9460-fed6-4e8d-8625-5915605902e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46b8e40d-h7bqq\" (UID: \"32fe9460-fed6-4e8d-8625-5915605902e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" Nov 02 20:32:13.600678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.600624 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fe9460-fed6-4e8d-8625-5915605902e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46b8e40d-h7bqq\" (UID: \"32fe9460-fed6-4e8d-8625-5915605902e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" Nov 02 20:32:13.601308 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.601289 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fe9460-fed6-4e8d-8625-5915605902e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46b8e40d-h7bqq\" (UID: \"32fe9460-fed6-4e8d-8625-5915605902e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" Nov 02 20:32:13.603192 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.603174 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fe9460-fed6-4e8d-8625-5915605902e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46b8e40d-h7bqq\" (UID: \"32fe9460-fed6-4e8d-8625-5915605902e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" Nov 02 20:32:13.648858 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.648832 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw"] Nov 02 20:32:13.658557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.658534 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pq4d\" (UniqueName: \"kubernetes.io/projected/32fe9460-fed6-4e8d-8625-5915605902e0-kube-api-access-2pq4d\") pod \"maas-api-revoke-sub-46b8e40d-h7bqq\" (UID: \"32fe9460-fed6-4e8d-8625-5915605902e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" Nov 02 20:32:13.682666 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.682637 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-g69pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:32:13.682859 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.682687 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" podUID="b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 20:32:13.715903 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.715845 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" Nov 02 20:32:13.970985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.970951 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" event={"ID":"10dd59b7-c045-4b1a-a27e-b7034e0b653d","Type":"ContainerStarted","Data":"3a776f5ba0e19fe7aada11d831ca6de2e2d921574c0e4d7a751487a5fddf0725"} Nov 02 20:32:13.971179 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:13.970997 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" event={"ID":"10dd59b7-c045-4b1a-a27e-b7034e0b653d","Type":"ContainerStarted","Data":"d83094d927e9e5b27133b5b7d6e479abb0bd18b366d4d276a81ce5f40db18d72"} Nov 02 20:32:14.084497 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.084399 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podStartSLOduration=2.084385378 podStartE2EDuration="2.084385378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:14.082482421 +0000 UTC m=+1062.372138950" watchObservedRunningTime="2026-11-02 20:32:14.084385378 +0000 UTC m=+1062.374041937" Nov 02 20:32:14.149107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.143483 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq"] Nov 02 20:32:14.152676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.152648 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm"] Nov 02 20:32:14.158456 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.158435 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm" Nov 02 20:32:14.205555 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.205528 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm"] Nov 02 20:32:14.208100 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.208050 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cebecc27-e1dd-4b63-bec3-20dbba31e58b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59586e36-q7kbm\" (UID: \"cebecc27-e1dd-4b63-bec3-20dbba31e58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm" Nov 02 20:32:14.208339 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.208295 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cebecc27-e1dd-4b63-bec3-20dbba31e58b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59586e36-q7kbm\" (UID: \"cebecc27-e1dd-4b63-bec3-20dbba31e58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm" Nov 02 20:32:14.208429 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.208413 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq7db\" (UniqueName: \"kubernetes.io/projected/cebecc27-e1dd-4b63-bec3-20dbba31e58b-kube-api-access-qq7db\") pod \"maas-api-revoke-sub-59586e36-q7kbm\" (UID: \"cebecc27-e1dd-4b63-bec3-20dbba31e58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm" Nov 02 20:32:14.257581 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.257547 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:14.309635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.309608 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cebecc27-e1dd-4b63-bec3-20dbba31e58b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59586e36-q7kbm\" (UID: \"cebecc27-e1dd-4b63-bec3-20dbba31e58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm" Nov 02 20:32:14.309782 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.309689 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq7db\" (UniqueName: \"kubernetes.io/projected/cebecc27-e1dd-4b63-bec3-20dbba31e58b-kube-api-access-qq7db\") pod \"maas-api-revoke-sub-59586e36-q7kbm\" (UID: \"cebecc27-e1dd-4b63-bec3-20dbba31e58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm" Nov 02 20:32:14.309828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.309818 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cebecc27-e1dd-4b63-bec3-20dbba31e58b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59586e36-q7kbm\" (UID: \"cebecc27-e1dd-4b63-bec3-20dbba31e58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm" Nov 02 20:32:14.310347 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.310329 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cebecc27-e1dd-4b63-bec3-20dbba31e58b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59586e36-q7kbm\" (UID: \"cebecc27-e1dd-4b63-bec3-20dbba31e58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm" Nov 02 20:32:14.348154 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.345486 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cebecc27-e1dd-4b63-bec3-20dbba31e58b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59586e36-q7kbm\" (UID: \"cebecc27-e1dd-4b63-bec3-20dbba31e58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm" Nov 02 20:32:14.367698 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.367671 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq7db\" (UniqueName: \"kubernetes.io/projected/cebecc27-e1dd-4b63-bec3-20dbba31e58b-kube-api-access-qq7db\") pod \"maas-api-revoke-sub-59586e36-q7kbm\" (UID: \"cebecc27-e1dd-4b63-bec3-20dbba31e58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm" Nov 02 20:32:14.483851 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.483814 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm" Nov 02 20:32:14.683046 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.682973 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-g69pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:32:14.683046 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.683026 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" podUID="b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 20:32:14.878745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.878702 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm"] Nov 02 20:32:14.879127 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:32:14.878903 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebecc27_e1dd_4b63_bec3_20dbba31e58b.slice/crio-7bc30232cc4a8c3ca428b707dfe30f9769a57742d1bc32e9a88a6cbce4c158f7 WatchSource:0}: Error finding container 7bc30232cc4a8c3ca428b707dfe30f9769a57742d1bc32e9a88a6cbce4c158f7: Status 404 returned error can't find the container with id 7bc30232cc4a8c3ca428b707dfe30f9769a57742d1bc32e9a88a6cbce4c158f7 Nov 02 20:32:14.979838 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.979802 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" event={"ID":"32fe9460-fed6-4e8d-8625-5915605902e0","Type":"ContainerStarted","Data":"c5638dbccc939ba069356f1fb328360089e4864d4c8f10fba549e4b6614cf523"} Nov 02 20:32:14.979978 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.979870 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" event={"ID":"32fe9460-fed6-4e8d-8625-5915605902e0","Type":"ContainerStarted","Data":"63708aa8d31675c2f31f3b132f36c319073e0515bab6a103eb98b39e9bbe4201"} Nov 02 20:32:14.981438 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:14.981411 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm" event={"ID":"cebecc27-e1dd-4b63-bec3-20dbba31e58b","Type":"ContainerStarted","Data":"7bc30232cc4a8c3ca428b707dfe30f9769a57742d1bc32e9a88a6cbce4c158f7"} Nov 02 20:32:15.094252 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:15.094202 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" podStartSLOduration=2.094179439 podStartE2EDuration="2.094179439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:15.093409067 +0000 UTC m=+1063.383065595" watchObservedRunningTime="2026-11-02 20:32:15.094179439 +0000 UTC m=+1063.383835985" Nov 02 20:32:15.683327 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:15.683296 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-g69pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:32:15.683565 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:15.683345 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" podUID="b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 20:32:16.006365 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:16.006332 4971 generic.go:356] "Generic (PLEG): container finished" podID="cebecc27-e1dd-4b63-bec3-20dbba31e58b" containerID="7b6107a0f0d2fb05050ee3ddf351c30022fdcd28fa45f95277098109c90782aa" exitCode=0 Nov 02 20:32:16.006734 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:16.006422 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm" event={"ID":"cebecc27-e1dd-4b63-bec3-20dbba31e58b","Type":"ContainerDied","Data":"7b6107a0f0d2fb05050ee3ddf351c30022fdcd28fa45f95277098109c90782aa"} Nov 02 20:32:16.008229 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:16.008208 4971 generic.go:356] "Generic (PLEG): container finished" podID="32fe9460-fed6-4e8d-8625-5915605902e0" containerID="c5638dbccc939ba069356f1fb328360089e4864d4c8f10fba549e4b6614cf523" exitCode=0 Nov 02 20:32:16.008342 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:16.008287 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" event={"ID":"32fe9460-fed6-4e8d-8625-5915605902e0","Type":"ContainerDied","Data":"c5638dbccc939ba069356f1fb328360089e4864d4c8f10fba549e4b6614cf523"} Nov 02 20:32:16.683225 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:16.683189 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-g69pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:32:16.683396 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:16.683244 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" podUID="b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 20:32:17.233019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.232994 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm" Nov 02 20:32:17.236517 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.236496 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" Nov 02 20:32:17.345005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.343969 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32fe9460-fed6-4e8d-8625-5915605902e0-kube-api-access-2pq4d\" (UniqueName: \"kubernetes.io/projected/32fe9460-fed6-4e8d-8625-5915605902e0-kube-api-access-2pq4d\") pod \"32fe9460-fed6-4e8d-8625-5915605902e0\" (UID: \"32fe9460-fed6-4e8d-8625-5915605902e0\") " Nov 02 20:32:17.345005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.344052 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cebecc27-e1dd-4b63-bec3-20dbba31e58b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cebecc27-e1dd-4b63-bec3-20dbba31e58b-maas-api-xks-ca\") pod \"cebecc27-e1dd-4b63-bec3-20dbba31e58b\" (UID: \"cebecc27-e1dd-4b63-bec3-20dbba31e58b\") " Nov 02 20:32:17.345005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.344128 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cebecc27-e1dd-4b63-bec3-20dbba31e58b-kube-api-access-qq7db\" (UniqueName: \"kubernetes.io/projected/cebecc27-e1dd-4b63-bec3-20dbba31e58b-kube-api-access-qq7db\") pod \"cebecc27-e1dd-4b63-bec3-20dbba31e58b\" (UID: \"cebecc27-e1dd-4b63-bec3-20dbba31e58b\") " Nov 02 20:32:17.345005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.344201 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32fe9460-fed6-4e8d-8625-5915605902e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fe9460-fed6-4e8d-8625-5915605902e0-maas-api-xks-ca\") pod \"32fe9460-fed6-4e8d-8625-5915605902e0\" (UID: \"32fe9460-fed6-4e8d-8625-5915605902e0\") " Nov 02 20:32:17.345005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.344330 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32fe9460-fed6-4e8d-8625-5915605902e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fe9460-fed6-4e8d-8625-5915605902e0-maas-api-openshift-ca\") pod \"32fe9460-fed6-4e8d-8625-5915605902e0\" (UID: \"32fe9460-fed6-4e8d-8625-5915605902e0\") " Nov 02 20:32:17.345005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.344380 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cebecc27-e1dd-4b63-bec3-20dbba31e58b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cebecc27-e1dd-4b63-bec3-20dbba31e58b-maas-api-openshift-ca\") pod \"cebecc27-e1dd-4b63-bec3-20dbba31e58b\" (UID: \"cebecc27-e1dd-4b63-bec3-20dbba31e58b\") " Nov 02 20:32:17.346445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.345230 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cebecc27-e1dd-4b63-bec3-20dbba31e58b-maas-api-openshift-ca" pod "cebecc27-e1dd-4b63-bec3-20dbba31e58b" (UID: "cebecc27-e1dd-4b63-bec3-20dbba31e58b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:17.346953 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.346900 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32fe9460-fed6-4e8d-8625-5915605902e0-maas-api-openshift-ca" pod "32fe9460-fed6-4e8d-8625-5915605902e0" (UID: "32fe9460-fed6-4e8d-8625-5915605902e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:17.349450 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.349423 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cebecc27-e1dd-4b63-bec3-20dbba31e58b-kube-api-access-qq7db" pod "cebecc27-e1dd-4b63-bec3-20dbba31e58b" (UID: "cebecc27-e1dd-4b63-bec3-20dbba31e58b"). InnerVolumeSpecName "kube-api-access-qq7db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:17.349818 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.349790 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cebecc27-e1dd-4b63-bec3-20dbba31e58b-maas-api-xks-ca" pod "cebecc27-e1dd-4b63-bec3-20dbba31e58b" (UID: "cebecc27-e1dd-4b63-bec3-20dbba31e58b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:17.349995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.349848 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32fe9460-fed6-4e8d-8625-5915605902e0-maas-api-xks-ca" pod "32fe9460-fed6-4e8d-8625-5915605902e0" (UID: "32fe9460-fed6-4e8d-8625-5915605902e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:17.349995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.349938 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fe9460-fed6-4e8d-8625-5915605902e0-kube-api-access-2pq4d" pod "32fe9460-fed6-4e8d-8625-5915605902e0" (UID: "32fe9460-fed6-4e8d-8625-5915605902e0"). InnerVolumeSpecName "kube-api-access-2pq4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:17.445745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.445701 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pq4d\" (UniqueName: \"kubernetes.io/projected/32fe9460-fed6-4e8d-8625-5915605902e0-kube-api-access-2pq4d\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:17.445745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.445733 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cebecc27-e1dd-4b63-bec3-20dbba31e58b-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:17.445745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.445748 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq7db\" (UniqueName: \"kubernetes.io/projected/cebecc27-e1dd-4b63-bec3-20dbba31e58b-kube-api-access-qq7db\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:17.446042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.445757 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fe9460-fed6-4e8d-8625-5915605902e0-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:17.446042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.445766 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fe9460-fed6-4e8d-8625-5915605902e0-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:17.446042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:17.445776 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cebecc27-e1dd-4b63-bec3-20dbba31e58b-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:18.026001 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:18.025962 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm" event={"ID":"cebecc27-e1dd-4b63-bec3-20dbba31e58b","Type":"ContainerDied","Data":"7bc30232cc4a8c3ca428b707dfe30f9769a57742d1bc32e9a88a6cbce4c158f7"} Nov 02 20:32:18.026196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:18.026039 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bc30232cc4a8c3ca428b707dfe30f9769a57742d1bc32e9a88a6cbce4c158f7" Nov 02 20:32:18.026196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:18.025984 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm" Nov 02 20:32:18.027794 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:18.027769 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" event={"ID":"32fe9460-fed6-4e8d-8625-5915605902e0","Type":"ContainerDied","Data":"63708aa8d31675c2f31f3b132f36c319073e0515bab6a103eb98b39e9bbe4201"} Nov 02 20:32:18.027913 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:18.027815 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63708aa8d31675c2f31f3b132f36c319073e0515bab6a103eb98b39e9bbe4201" Nov 02 20:32:18.027913 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:18.027833 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq" Nov 02 20:32:18.257745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:18.257624 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:18.723216 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:18.723185 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-g69pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:32:18.723432 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:18.723233 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" podUID="b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:32:19.690880 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:19.690856 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:19.691322 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:19.691299 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:19.692123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:19.692072 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:32:22.023816 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.023780 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk"] Nov 02 20:32:22.024591 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.024569 4971 cpu_manager.go:414] "containerMap: removing container" podUID="32fe9460-fed6-4e8d-8625-5915605902e0" containerName="revoke-keys" Nov 02 20:32:22.024699 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.024593 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32fe9460-fed6-4e8d-8625-5915605902e0" containerName="revoke-keys" Nov 02 20:32:22.024699 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.024622 4971 cpu_manager.go:414] "containerMap: removing container" podUID="cebecc27-e1dd-4b63-bec3-20dbba31e58b" containerName="revoke-keys" Nov 02 20:32:22.024699 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.024630 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cebecc27-e1dd-4b63-bec3-20dbba31e58b" containerName="revoke-keys" Nov 02 20:32:22.024837 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.024757 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="32fe9460-fed6-4e8d-8625-5915605902e0" containerName="revoke-keys" Nov 02 20:32:22.024837 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.024776 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="cebecc27-e1dd-4b63-bec3-20dbba31e58b" containerName="revoke-keys" Nov 02 20:32:22.029482 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.029460 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" Nov 02 20:32:22.082788 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.082758 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk"] Nov 02 20:32:22.091884 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.091858 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:32:22.107821 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.107793 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg7q2\" (UniqueName: \"kubernetes.io/projected/0576b7e9-c351-4260-86bf-10e795f98f46-kube-api-access-zg7q2\") pod \"maas-api-revoke-sub-6539a1ee-lcsxk\" (UID: \"0576b7e9-c351-4260-86bf-10e795f98f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" Nov 02 20:32:22.107926 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.107857 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0576b7e9-c351-4260-86bf-10e795f98f46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6539a1ee-lcsxk\" (UID: \"0576b7e9-c351-4260-86bf-10e795f98f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" Nov 02 20:32:22.108031 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.108009 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0576b7e9-c351-4260-86bf-10e795f98f46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6539a1ee-lcsxk\" (UID: \"0576b7e9-c351-4260-86bf-10e795f98f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" Nov 02 20:32:22.124065 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.124048 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:32:22.209283 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.209260 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg7q2\" (UniqueName: \"kubernetes.io/projected/0576b7e9-c351-4260-86bf-10e795f98f46-kube-api-access-zg7q2\") pod \"maas-api-revoke-sub-6539a1ee-lcsxk\" (UID: \"0576b7e9-c351-4260-86bf-10e795f98f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" Nov 02 20:32:22.209391 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.209302 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0576b7e9-c351-4260-86bf-10e795f98f46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6539a1ee-lcsxk\" (UID: \"0576b7e9-c351-4260-86bf-10e795f98f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" Nov 02 20:32:22.209391 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.209347 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0576b7e9-c351-4260-86bf-10e795f98f46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6539a1ee-lcsxk\" (UID: \"0576b7e9-c351-4260-86bf-10e795f98f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" Nov 02 20:32:22.209915 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.209895 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0576b7e9-c351-4260-86bf-10e795f98f46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6539a1ee-lcsxk\" (UID: \"0576b7e9-c351-4260-86bf-10e795f98f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" Nov 02 20:32:22.211847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.211825 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0576b7e9-c351-4260-86bf-10e795f98f46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6539a1ee-lcsxk\" (UID: \"0576b7e9-c351-4260-86bf-10e795f98f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" Nov 02 20:32:22.266860 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.266838 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg7q2\" (UniqueName: \"kubernetes.io/projected/0576b7e9-c351-4260-86bf-10e795f98f46-kube-api-access-zg7q2\") pod \"maas-api-revoke-sub-6539a1ee-lcsxk\" (UID: \"0576b7e9-c351-4260-86bf-10e795f98f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" Nov 02 20:32:22.343013 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.342963 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" Nov 02 20:32:22.712380 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:32:22.712348 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0576b7e9_c351_4260_86bf_10e795f98f46.slice/crio-317ca96f03b5d4222645c2868bc63d369254489621539cc868d92a9fc74c0e27 WatchSource:0}: Error finding container 317ca96f03b5d4222645c2868bc63d369254489621539cc868d92a9fc74c0e27: Status 404 returned error can't find the container with id 317ca96f03b5d4222645c2868bc63d369254489621539cc868d92a9fc74c0e27 Nov 02 20:32:22.712806 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:22.712781 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk"] Nov 02 20:32:23.069351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.069320 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" event={"ID":"0576b7e9-c351-4260-86bf-10e795f98f46","Type":"ContainerStarted","Data":"f176053d6f6b245ffc5bddecdbf489b8ce7ed444b2de66713c88c6b8bc17d71a"} Nov 02 20:32:23.069734 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.069384 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" event={"ID":"0576b7e9-c351-4260-86bf-10e795f98f46","Type":"ContainerStarted","Data":"317ca96f03b5d4222645c2868bc63d369254489621539cc868d92a9fc74c0e27"} Nov 02 20:32:23.179784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.179729 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" podStartSLOduration=2.179714674 podStartE2EDuration="2.179714674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:23.17821535 +0000 UTC m=+1071.467871881" watchObservedRunningTime="2026-11-02 20:32:23.179714674 +0000 UTC m=+1071.469371263" Nov 02 20:32:23.236681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.236643 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:23.242317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.242292 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:23.255883 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.255862 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:23.381338 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.381275 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv"] Nov 02 20:32:23.386853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.386832 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.407421 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.407393 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt"] Nov 02 20:32:23.412394 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.412368 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.444025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.443993 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv"] Nov 02 20:32:23.498207 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.498158 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt"] Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.540763 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.540824 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkm5w\" (UniqueName: \"kubernetes.io/projected/817f7e35-0db0-493e-a3af-825e1818eb18-kube-api-access-jkm5w\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.540863 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.540896 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.540925 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02d7838-6668-43b5-b0d8-4e5af366edf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.540953 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.540975 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.541025 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.541095 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/817f7e35-0db0-493e-a3af-825e1818eb18-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.541149 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.541184 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.541246 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/817f7e35-0db0-493e-a3af-825e1818eb18-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.541281 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.541333 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.541385 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.541439 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/817f7e35-0db0-493e-a3af-825e1818eb18-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.541477 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm8ng\" (UniqueName: \"kubernetes.io/projected/a02d7838-6668-43b5-b0d8-4e5af366edf1-kube-api-access-cm8ng\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.542993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.541514 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.665498 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.665157 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.666410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.666385 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02d7838-6668-43b5-b0d8-4e5af366edf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.673038 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.673003 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.673172 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.673055 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.673172 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.673131 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.673172 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.673157 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/817f7e35-0db0-493e-a3af-825e1818eb18-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.673353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.673212 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.673353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.673247 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.673353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.673313 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/817f7e35-0db0-493e-a3af-825e1818eb18-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.673353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.673351 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.673572 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.673407 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.673572 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.673462 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.673572 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.673516 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/817f7e35-0db0-493e-a3af-825e1818eb18-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.673572 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.673559 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm8ng\" (UniqueName: \"kubernetes.io/projected/a02d7838-6668-43b5-b0d8-4e5af366edf1-kube-api-access-cm8ng\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.673793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.673599 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.673793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.673679 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.673793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.673721 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkm5w\" (UniqueName: \"kubernetes.io/projected/817f7e35-0db0-493e-a3af-825e1818eb18-kube-api-access-jkm5w\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.673793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.673765 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.674248 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.674223 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.674358 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.667416 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02d7838-6668-43b5-b0d8-4e5af366edf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.674895 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.674518 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.677320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.675526 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/817f7e35-0db0-493e-a3af-825e1818eb18-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.677320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.666170 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.677320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.676059 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.677320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.676332 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.677320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.676869 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.677320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.677062 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.677320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.677248 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.683308 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.683286 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.688558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.688486 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.689488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.689423 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/817f7e35-0db0-493e-a3af-825e1818eb18-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.701628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.701600 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.749426 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.749357 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.751178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.751128 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm8ng\" (UniqueName: \"kubernetes.io/projected/a02d7838-6668-43b5-b0d8-4e5af366edf1-kube-api-access-cm8ng\") pod \"maas-default-gateway-openshift-default-85544d4cd7-xx5bv\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:23.768865 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.768656 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkm5w\" (UniqueName: \"kubernetes.io/projected/817f7e35-0db0-493e-a3af-825e1818eb18-kube-api-access-jkm5w\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:23.775648 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:23.775617 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/817f7e35-0db0-493e-a3af-825e1818eb18-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gr2jt\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:24.005421 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:24.005324 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:24.037099 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:24.036293 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:24.200190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:24.198743 4971 generic.go:356] "Generic (PLEG): container finished" podID="0576b7e9-c351-4260-86bf-10e795f98f46" containerID="f176053d6f6b245ffc5bddecdbf489b8ce7ed444b2de66713c88c6b8bc17d71a" exitCode=0 Nov 02 20:32:24.200190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:24.199185 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" event={"ID":"0576b7e9-c351-4260-86bf-10e795f98f46","Type":"ContainerDied","Data":"f176053d6f6b245ffc5bddecdbf489b8ce7ed444b2de66713c88c6b8bc17d71a"} Nov 02 20:32:24.214647 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:24.214620 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:32:24.536108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:24.522272 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv"] Nov 02 20:32:24.559591 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:24.556616 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:24.559591 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:24.556736 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:24.559591 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:24.556802 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:24.611334 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:24.611307 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt"] Nov 02 20:32:24.626055 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:32:24.626009 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817f7e35_0db0_493e_a3af_825e1818eb18.slice/crio-c23460676d979890d53df175e60582a962856d64b45b348fd31bacc31642e16c WatchSource:0}: Error finding container c23460676d979890d53df175e60582a962856d64b45b348fd31bacc31642e16c: Status 404 returned error can't find the container with id c23460676d979890d53df175e60582a962856d64b45b348fd31bacc31642e16c Nov 02 20:32:24.628917 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:24.628881 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:24.629031 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:24.628965 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:24.629031 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:24.629014 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:25.069337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.069302 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8"] Nov 02 20:32:25.073510 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.073491 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8" Nov 02 20:32:25.130630 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.130598 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8"] Nov 02 20:32:25.155396 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.155364 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f630eb29-efe9-490c-b8e5-8ecc7de298cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11a26d24-sbrp8\" (UID: \"f630eb29-efe9-490c-b8e5-8ecc7de298cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8" Nov 02 20:32:25.155396 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.155399 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5jp8\" (UniqueName: \"kubernetes.io/projected/f630eb29-efe9-490c-b8e5-8ecc7de298cc-kube-api-access-k5jp8\") pod \"maas-api-revoke-sub-11a26d24-sbrp8\" (UID: \"f630eb29-efe9-490c-b8e5-8ecc7de298cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8" Nov 02 20:32:25.155626 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.155615 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f630eb29-efe9-490c-b8e5-8ecc7de298cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11a26d24-sbrp8\" (UID: \"f630eb29-efe9-490c-b8e5-8ecc7de298cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8" Nov 02 20:32:25.276255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.276208 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" event={"ID":"817f7e35-0db0-493e-a3af-825e1818eb18","Type":"ContainerStarted","Data":"3052b3b7f6606fb2fd26bc1c17f042031c0fbcd9aa74be8ef86a3987798375eb"} Nov 02 20:32:25.276878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.276834 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" event={"ID":"817f7e35-0db0-493e-a3af-825e1818eb18","Type":"ContainerStarted","Data":"c23460676d979890d53df175e60582a962856d64b45b348fd31bacc31642e16c"} Nov 02 20:32:25.281891 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.280894 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f630eb29-efe9-490c-b8e5-8ecc7de298cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11a26d24-sbrp8\" (UID: \"f630eb29-efe9-490c-b8e5-8ecc7de298cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8" Nov 02 20:32:25.281891 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.281162 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f630eb29-efe9-490c-b8e5-8ecc7de298cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11a26d24-sbrp8\" (UID: \"f630eb29-efe9-490c-b8e5-8ecc7de298cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8" Nov 02 20:32:25.281891 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.281193 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5jp8\" (UniqueName: \"kubernetes.io/projected/f630eb29-efe9-490c-b8e5-8ecc7de298cc-kube-api-access-k5jp8\") pod \"maas-api-revoke-sub-11a26d24-sbrp8\" (UID: \"f630eb29-efe9-490c-b8e5-8ecc7de298cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8" Nov 02 20:32:25.283231 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.283203 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f630eb29-efe9-490c-b8e5-8ecc7de298cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11a26d24-sbrp8\" (UID: \"f630eb29-efe9-490c-b8e5-8ecc7de298cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8" Nov 02 20:32:25.283917 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.283892 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" event={"ID":"a02d7838-6668-43b5-b0d8-4e5af366edf1","Type":"ContainerStarted","Data":"aba37b744deaecfe6f274adca6819cd64d018045121794e71f96078a8bfd05b5"} Nov 02 20:32:25.284013 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.283965 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" event={"ID":"a02d7838-6668-43b5-b0d8-4e5af366edf1","Type":"ContainerStarted","Data":"7d0c91cc2731bb7a06aa2abae54db61905b79792bfba94a597b4de2d281ad08c"} Nov 02 20:32:25.286434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.286415 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f630eb29-efe9-490c-b8e5-8ecc7de298cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11a26d24-sbrp8\" (UID: \"f630eb29-efe9-490c-b8e5-8ecc7de298cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8" Nov 02 20:32:25.374926 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.374857 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5jp8\" (UniqueName: \"kubernetes.io/projected/f630eb29-efe9-490c-b8e5-8ecc7de298cc-kube-api-access-k5jp8\") pod \"maas-api-revoke-sub-11a26d24-sbrp8\" (UID: \"f630eb29-efe9-490c-b8e5-8ecc7de298cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8" Nov 02 20:32:25.390649 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.390599 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8" Nov 02 20:32:25.398212 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.397847 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" podStartSLOduration=2.3978147659999998 podStartE2EDuration="2.397814766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:25.396558078 +0000 UTC m=+1073.686214608" watchObservedRunningTime="2026-11-02 20:32:25.397814766 +0000 UTC m=+1073.687471294" Nov 02 20:32:25.521871 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.521629 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" Nov 02 20:32:25.523641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.523583 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" podStartSLOduration=2.523565326 podStartE2EDuration="2.523565326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:25.514990799 +0000 UTC m=+1073.804647328" watchObservedRunningTime="2026-11-02 20:32:25.523565326 +0000 UTC m=+1073.813221855" Nov 02 20:32:25.688377 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.686400 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0576b7e9-c351-4260-86bf-10e795f98f46-kube-api-access-zg7q2\" (UniqueName: \"kubernetes.io/projected/0576b7e9-c351-4260-86bf-10e795f98f46-kube-api-access-zg7q2\") pod \"0576b7e9-c351-4260-86bf-10e795f98f46\" (UID: \"0576b7e9-c351-4260-86bf-10e795f98f46\") " Nov 02 20:32:25.688377 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.686444 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0576b7e9-c351-4260-86bf-10e795f98f46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0576b7e9-c351-4260-86bf-10e795f98f46-maas-api-openshift-ca\") pod \"0576b7e9-c351-4260-86bf-10e795f98f46\" (UID: \"0576b7e9-c351-4260-86bf-10e795f98f46\") " Nov 02 20:32:25.688377 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.686659 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0576b7e9-c351-4260-86bf-10e795f98f46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0576b7e9-c351-4260-86bf-10e795f98f46-maas-api-xks-ca\") pod \"0576b7e9-c351-4260-86bf-10e795f98f46\" (UID: \"0576b7e9-c351-4260-86bf-10e795f98f46\") " Nov 02 20:32:25.688377 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.687154 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0576b7e9-c351-4260-86bf-10e795f98f46-maas-api-openshift-ca" pod "0576b7e9-c351-4260-86bf-10e795f98f46" (UID: "0576b7e9-c351-4260-86bf-10e795f98f46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:25.689885 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.689836 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0576b7e9-c351-4260-86bf-10e795f98f46-maas-api-xks-ca" pod "0576b7e9-c351-4260-86bf-10e795f98f46" (UID: "0576b7e9-c351-4260-86bf-10e795f98f46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:25.692152 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.692093 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0576b7e9-c351-4260-86bf-10e795f98f46-kube-api-access-zg7q2" pod "0576b7e9-c351-4260-86bf-10e795f98f46" (UID: "0576b7e9-c351-4260-86bf-10e795f98f46"). InnerVolumeSpecName "kube-api-access-zg7q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:25.787966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.787930 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg7q2\" (UniqueName: \"kubernetes.io/projected/0576b7e9-c351-4260-86bf-10e795f98f46-kube-api-access-zg7q2\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:25.787966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.787963 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0576b7e9-c351-4260-86bf-10e795f98f46-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:25.788189 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.787975 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0576b7e9-c351-4260-86bf-10e795f98f46-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:25.900636 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:25.900610 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8"] Nov 02 20:32:26.006183 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:26.006097 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:26.007976 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:26.007949 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xx5bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:32:26.008148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:26.007997 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" podUID="a02d7838-6668-43b5-b0d8-4e5af366edf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 20:32:26.057108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:26.056266 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:26.057108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:26.056415 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gr2jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:32:26.057108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:26.056480 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" podUID="817f7e35-0db0-493e-a3af-825e1818eb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 20:32:26.292470 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:26.292438 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" event={"ID":"0576b7e9-c351-4260-86bf-10e795f98f46","Type":"ContainerDied","Data":"317ca96f03b5d4222645c2868bc63d369254489621539cc868d92a9fc74c0e27"} Nov 02 20:32:26.292832 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:26.292488 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="317ca96f03b5d4222645c2868bc63d369254489621539cc868d92a9fc74c0e27" Nov 02 20:32:26.292832 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:26.292551 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk" Nov 02 20:32:26.294690 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:26.294667 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8" event={"ID":"f630eb29-efe9-490c-b8e5-8ecc7de298cc","Type":"ContainerStarted","Data":"b3975ced806ab0664b483ac92cea0baef2cb8d9c5f461ca1cf48136dc236cd80"} Nov 02 20:32:27.006838 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:27.006807 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xx5bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:32:27.007019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:27.006857 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" podUID="a02d7838-6668-43b5-b0d8-4e5af366edf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 20:32:27.078132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:27.043416 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gr2jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:32:27.078132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:27.077194 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" podUID="817f7e35-0db0-493e-a3af-825e1818eb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 20:32:27.302781 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:27.302688 4971 generic.go:356] "Generic (PLEG): container finished" podID="f630eb29-efe9-490c-b8e5-8ecc7de298cc" containerID="1220ad89a97e422695ce494803b299c84d54331b7e3d25e1159cc2f5595d6e9f" exitCode=0 Nov 02 20:32:27.303164 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:27.302814 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8" event={"ID":"f630eb29-efe9-490c-b8e5-8ecc7de298cc","Type":"ContainerDied","Data":"1220ad89a97e422695ce494803b299c84d54331b7e3d25e1159cc2f5595d6e9f"} Nov 02 20:32:27.705179 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:27.705145 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:28.006780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:28.006695 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xx5bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:32:28.006780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:28.006747 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" podUID="a02d7838-6668-43b5-b0d8-4e5af366edf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 20:32:28.041108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:28.039735 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gr2jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:32:28.041108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:28.039789 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" podUID="817f7e35-0db0-493e-a3af-825e1818eb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 20:32:28.110297 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:28.110266 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:28.595941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:28.595911 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8" Nov 02 20:32:28.618931 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:28.618905 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f630eb29-efe9-490c-b8e5-8ecc7de298cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f630eb29-efe9-490c-b8e5-8ecc7de298cc-maas-api-xks-ca\") pod \"f630eb29-efe9-490c-b8e5-8ecc7de298cc\" (UID: \"f630eb29-efe9-490c-b8e5-8ecc7de298cc\") " Nov 02 20:32:28.619060 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:28.619013 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f630eb29-efe9-490c-b8e5-8ecc7de298cc-kube-api-access-k5jp8\" (UniqueName: \"kubernetes.io/projected/f630eb29-efe9-490c-b8e5-8ecc7de298cc-kube-api-access-k5jp8\") pod \"f630eb29-efe9-490c-b8e5-8ecc7de298cc\" (UID: \"f630eb29-efe9-490c-b8e5-8ecc7de298cc\") " Nov 02 20:32:28.619187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:28.619174 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f630eb29-efe9-490c-b8e5-8ecc7de298cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f630eb29-efe9-490c-b8e5-8ecc7de298cc-maas-api-openshift-ca\") pod \"f630eb29-efe9-490c-b8e5-8ecc7de298cc\" (UID: \"f630eb29-efe9-490c-b8e5-8ecc7de298cc\") " Nov 02 20:32:28.619525 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:28.619491 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f630eb29-efe9-490c-b8e5-8ecc7de298cc-maas-api-openshift-ca" pod "f630eb29-efe9-490c-b8e5-8ecc7de298cc" (UID: "f630eb29-efe9-490c-b8e5-8ecc7de298cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:28.621172 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:28.621150 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f630eb29-efe9-490c-b8e5-8ecc7de298cc-kube-api-access-k5jp8" pod "f630eb29-efe9-490c-b8e5-8ecc7de298cc" (UID: "f630eb29-efe9-490c-b8e5-8ecc7de298cc"). InnerVolumeSpecName "kube-api-access-k5jp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:28.621283 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:28.621265 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f630eb29-efe9-490c-b8e5-8ecc7de298cc-maas-api-xks-ca" pod "f630eb29-efe9-490c-b8e5-8ecc7de298cc" (UID: "f630eb29-efe9-490c-b8e5-8ecc7de298cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:28.720712 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:28.720677 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f630eb29-efe9-490c-b8e5-8ecc7de298cc-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:28.720712 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:28.720708 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f630eb29-efe9-490c-b8e5-8ecc7de298cc-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:28.720712 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:28.720718 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5jp8\" (UniqueName: \"kubernetes.io/projected/f630eb29-efe9-490c-b8e5-8ecc7de298cc-kube-api-access-k5jp8\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:29.006398 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.006361 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xx5bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:32:29.006567 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.006413 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" podUID="a02d7838-6668-43b5-b0d8-4e5af366edf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 20:32:29.054376 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.054344 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gr2jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:32:29.054607 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.054580 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" podUID="817f7e35-0db0-493e-a3af-825e1818eb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 20:32:29.319034 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.318959 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8" Nov 02 20:32:29.319034 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.318991 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8" event={"ID":"f630eb29-efe9-490c-b8e5-8ecc7de298cc","Type":"ContainerDied","Data":"b3975ced806ab0664b483ac92cea0baef2cb8d9c5f461ca1cf48136dc236cd80"} Nov 02 20:32:29.319297 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.319071 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3975ced806ab0664b483ac92cea0baef2cb8d9c5f461ca1cf48136dc236cd80" Nov 02 20:32:29.866108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.859819 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm"] Nov 02 20:32:29.866108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.864561 4971 cpu_manager.go:414] "containerMap: removing container" podUID="f630eb29-efe9-490c-b8e5-8ecc7de298cc" containerName="revoke-keys" Nov 02 20:32:29.866108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.864584 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f630eb29-efe9-490c-b8e5-8ecc7de298cc" containerName="revoke-keys" Nov 02 20:32:29.866108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.864608 4971 cpu_manager.go:414] "containerMap: removing container" podUID="0576b7e9-c351-4260-86bf-10e795f98f46" containerName="revoke-keys" Nov 02 20:32:29.866108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.864617 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0576b7e9-c351-4260-86bf-10e795f98f46" containerName="revoke-keys" Nov 02 20:32:29.866108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.864815 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="f630eb29-efe9-490c-b8e5-8ecc7de298cc" containerName="revoke-keys" Nov 02 20:32:29.866108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.864832 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="0576b7e9-c351-4260-86bf-10e795f98f46" containerName="revoke-keys" Nov 02 20:32:29.879920 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.879896 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" Nov 02 20:32:29.906273 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.906238 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm"] Nov 02 20:32:29.952229 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.952202 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:32:29.979006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.978972 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c659ab03-c476-47a4-91c9-0457bb538cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcfbe15b-kcjjm\" (UID: \"c659ab03-c476-47a4-91c9-0457bb538cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" Nov 02 20:32:29.979256 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.979236 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j2ss\" (UniqueName: \"kubernetes.io/projected/c659ab03-c476-47a4-91c9-0457bb538cd7-kube-api-access-6j2ss\") pod \"maas-api-revoke-sub-fcfbe15b-kcjjm\" (UID: \"c659ab03-c476-47a4-91c9-0457bb538cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" Nov 02 20:32:29.979704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.979680 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c659ab03-c476-47a4-91c9-0457bb538cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcfbe15b-kcjjm\" (UID: \"c659ab03-c476-47a4-91c9-0457bb538cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" Nov 02 20:32:29.979910 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:29.979483 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:32:30.012239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:30.012209 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xx5bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:32:30.012469 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:30.012442 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" podUID="a02d7838-6668-43b5-b0d8-4e5af366edf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 20:32:30.039436 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:30.039403 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gr2jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:32:30.039674 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:30.039650 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" podUID="817f7e35-0db0-493e-a3af-825e1818eb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 20:32:30.081439 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:30.081407 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c659ab03-c476-47a4-91c9-0457bb538cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcfbe15b-kcjjm\" (UID: \"c659ab03-c476-47a4-91c9-0457bb538cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" Nov 02 20:32:30.081565 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:30.081450 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j2ss\" (UniqueName: \"kubernetes.io/projected/c659ab03-c476-47a4-91c9-0457bb538cd7-kube-api-access-6j2ss\") pod \"maas-api-revoke-sub-fcfbe15b-kcjjm\" (UID: \"c659ab03-c476-47a4-91c9-0457bb538cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" Nov 02 20:32:30.081565 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:30.081544 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c659ab03-c476-47a4-91c9-0457bb538cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcfbe15b-kcjjm\" (UID: \"c659ab03-c476-47a4-91c9-0457bb538cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" Nov 02 20:32:30.082290 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:30.082269 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c659ab03-c476-47a4-91c9-0457bb538cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcfbe15b-kcjjm\" (UID: \"c659ab03-c476-47a4-91c9-0457bb538cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" Nov 02 20:32:30.084362 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:30.084342 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c659ab03-c476-47a4-91c9-0457bb538cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcfbe15b-kcjjm\" (UID: \"c659ab03-c476-47a4-91c9-0457bb538cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" Nov 02 20:32:30.173255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:30.173182 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j2ss\" (UniqueName: \"kubernetes.io/projected/c659ab03-c476-47a4-91c9-0457bb538cd7-kube-api-access-6j2ss\") pod \"maas-api-revoke-sub-fcfbe15b-kcjjm\" (UID: \"c659ab03-c476-47a4-91c9-0457bb538cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" Nov 02 20:32:30.194883 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:30.194860 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" Nov 02 20:32:30.647119 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:30.647052 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm"] Nov 02 20:32:30.673412 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:32:30.672526 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc659ab03_c476_47a4_91c9_0457bb538cd7.slice/crio-a84b4f81cfa3c3a3cd1cca45668dace90af9cacba81a45c28c3ee71a0515afe8 WatchSource:0}: Error finding container a84b4f81cfa3c3a3cd1cca45668dace90af9cacba81a45c28c3ee71a0515afe8: Status 404 returned error can't find the container with id a84b4f81cfa3c3a3cd1cca45668dace90af9cacba81a45c28c3ee71a0515afe8 Nov 02 20:32:31.006326 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:31.006292 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xx5bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:32:31.006645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:31.006353 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" podUID="a02d7838-6668-43b5-b0d8-4e5af366edf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 20:32:31.037752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:31.037724 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gr2jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:32:31.037871 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:31.037779 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" podUID="817f7e35-0db0-493e-a3af-825e1818eb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 20:32:31.386442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:31.386400 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" event={"ID":"c659ab03-c476-47a4-91c9-0457bb538cd7","Type":"ContainerStarted","Data":"2ad3fcf831cfa6d9b204bc17cf21437033d1ae63b5c4e0f17ffdea8b43ec21b3"} Nov 02 20:32:31.386613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:31.386469 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" event={"ID":"c659ab03-c476-47a4-91c9-0457bb538cd7","Type":"ContainerStarted","Data":"a84b4f81cfa3c3a3cd1cca45668dace90af9cacba81a45c28c3ee71a0515afe8"} Nov 02 20:32:31.499331 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:31.499270 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" podStartSLOduration=2.4992514359999998 podStartE2EDuration="2.499251436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:31.495140268 +0000 UTC m=+1079.784796797" watchObservedRunningTime="2026-11-02 20:32:31.499251436 +0000 UTC m=+1079.788907977" Nov 02 20:32:32.006391 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:32.006306 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-xx5bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:32:32.006391 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:32.006364 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" podUID="a02d7838-6668-43b5-b0d8-4e5af366edf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 02 20:32:32.039514 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:32.039465 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gr2jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:32:32.039683 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:32.039543 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" podUID="817f7e35-0db0-493e-a3af-825e1818eb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 20:32:32.399336 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:32.399295 4971 generic.go:356] "Generic (PLEG): container finished" podID="c659ab03-c476-47a4-91c9-0457bb538cd7" containerID="2ad3fcf831cfa6d9b204bc17cf21437033d1ae63b5c4e0f17ffdea8b43ec21b3" exitCode=0 Nov 02 20:32:32.399665 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:32.399388 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" event={"ID":"c659ab03-c476-47a4-91c9-0457bb538cd7","Type":"ContainerDied","Data":"2ad3fcf831cfa6d9b204bc17cf21437033d1ae63b5c4e0f17ffdea8b43ec21b3"} Nov 02 20:32:32.938669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:32.938630 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:33.158518 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:33.158487 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:33.159012 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:33.158997 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:33.159982 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:33.159961 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:32:33.192480 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:33.192400 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:33.192828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:33.192805 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:33.193570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:33.193552 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:32:33.338317 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:32:33.338281 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:32:33.587844 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:33.587816 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" Nov 02 20:32:33.729381 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:33.729295 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c659ab03-c476-47a4-91c9-0457bb538cd7-kube-api-access-6j2ss\" (UniqueName: \"kubernetes.io/projected/c659ab03-c476-47a4-91c9-0457bb538cd7-kube-api-access-6j2ss\") pod \"c659ab03-c476-47a4-91c9-0457bb538cd7\" (UID: \"c659ab03-c476-47a4-91c9-0457bb538cd7\") " Nov 02 20:32:33.729538 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:33.729507 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c659ab03-c476-47a4-91c9-0457bb538cd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c659ab03-c476-47a4-91c9-0457bb538cd7-maas-api-xks-ca\") pod \"c659ab03-c476-47a4-91c9-0457bb538cd7\" (UID: \"c659ab03-c476-47a4-91c9-0457bb538cd7\") " Nov 02 20:32:33.729600 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:33.729551 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c659ab03-c476-47a4-91c9-0457bb538cd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c659ab03-c476-47a4-91c9-0457bb538cd7-maas-api-openshift-ca\") pod \"c659ab03-c476-47a4-91c9-0457bb538cd7\" (UID: \"c659ab03-c476-47a4-91c9-0457bb538cd7\") " Nov 02 20:32:33.730036 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:33.729955 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c659ab03-c476-47a4-91c9-0457bb538cd7-maas-api-openshift-ca" pod "c659ab03-c476-47a4-91c9-0457bb538cd7" (UID: "c659ab03-c476-47a4-91c9-0457bb538cd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:33.732139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:33.732111 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c659ab03-c476-47a4-91c9-0457bb538cd7-kube-api-access-6j2ss" pod "c659ab03-c476-47a4-91c9-0457bb538cd7" (UID: "c659ab03-c476-47a4-91c9-0457bb538cd7"). InnerVolumeSpecName "kube-api-access-6j2ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:33.735218 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:33.735189 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c659ab03-c476-47a4-91c9-0457bb538cd7-maas-api-xks-ca" pod "c659ab03-c476-47a4-91c9-0457bb538cd7" (UID: "c659ab03-c476-47a4-91c9-0457bb538cd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:33.830682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:33.830650 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c659ab03-c476-47a4-91c9-0457bb538cd7-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:33.830682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:33.830679 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c659ab03-c476-47a4-91c9-0457bb538cd7-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:33.830682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:33.830688 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j2ss\" (UniqueName: \"kubernetes.io/projected/c659ab03-c476-47a4-91c9-0457bb538cd7-kube-api-access-6j2ss\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:34.415767 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:34.415737 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" Nov 02 20:32:34.416156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:34.415771 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm" event={"ID":"c659ab03-c476-47a4-91c9-0457bb538cd7","Type":"ContainerDied","Data":"a84b4f81cfa3c3a3cd1cca45668dace90af9cacba81a45c28c3ee71a0515afe8"} Nov 02 20:32:34.416156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:34.415818 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a84b4f81cfa3c3a3cd1cca45668dace90af9cacba81a45c28c3ee71a0515afe8" Nov 02 20:32:35.241450 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.241406 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v"] Nov 02 20:32:35.242109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.242093 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c659ab03-c476-47a4-91c9-0457bb538cd7" containerName="revoke-keys" Nov 02 20:32:35.242178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.242110 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c659ab03-c476-47a4-91c9-0457bb538cd7" containerName="revoke-keys" Nov 02 20:32:35.242239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.242228 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c659ab03-c476-47a4-91c9-0457bb538cd7" containerName="revoke-keys" Nov 02 20:32:35.246146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.246127 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.298632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.298610 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-61cdc1-openshift-default-dockercfg-4zg5g\"" Nov 02 20:32:35.345757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.345723 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a943a64-646e-4533-a37c-71334481ec3a-istio-podinfo\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.345923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.345782 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-istio-envoy\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.345923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.345881 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-workload-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.346066 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.346038 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-istio-data\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.346229 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.346210 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-credential-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.346316 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.346248 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a943a64-646e-4533-a37c-71334481ec3a-istio-token\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.346316 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.346289 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87bj7\" (UniqueName: \"kubernetes.io/projected/7a943a64-646e-4533-a37c-71334481ec3a-kube-api-access-87bj7\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.346433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.346389 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-workload-certs\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.346433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.346415 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a943a64-646e-4533-a37c-71334481ec3a-istiod-ca-cert\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.349793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.349776 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v"] Nov 02 20:32:35.447047 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.447019 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a943a64-646e-4533-a37c-71334481ec3a-istio-token\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.447451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.447057 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87bj7\" (UniqueName: \"kubernetes.io/projected/7a943a64-646e-4533-a37c-71334481ec3a-kube-api-access-87bj7\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.447451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.447142 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-workload-certs\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.447451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.447163 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a943a64-646e-4533-a37c-71334481ec3a-istiod-ca-cert\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.447451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.447217 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a943a64-646e-4533-a37c-71334481ec3a-istio-podinfo\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.447451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.447257 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-istio-envoy\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.447451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.447288 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-workload-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.447451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.447428 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-istio-data\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.447836 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.447481 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-credential-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.447836 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.447566 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-workload-certs\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.447836 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.447724 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-workload-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.447836 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.447798 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-credential-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.448025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.447867 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-istio-data\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.448025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.447955 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a943a64-646e-4533-a37c-71334481ec3a-istiod-ca-cert\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.449640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.449619 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-istio-envoy\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.449862 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.449845 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a943a64-646e-4533-a37c-71334481ec3a-istio-podinfo\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.510884 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.510814 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a943a64-646e-4533-a37c-71334481ec3a-istio-token\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.511016 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.510939 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87bj7\" (UniqueName: \"kubernetes.io/projected/7a943a64-646e-4533-a37c-71334481ec3a-kube-api-access-87bj7\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.559227 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.559204 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:35.979813 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:32:35.979779 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a943a64_646e_4533_a37c_71334481ec3a.slice/crio-21c6277bb43c66fce9a1755c964b39f954f7bd37307badc67de329a8a9341069 WatchSource:0}: Error finding container 21c6277bb43c66fce9a1755c964b39f954f7bd37307badc67de329a8a9341069: Status 404 returned error can't find the container with id 21c6277bb43c66fce9a1755c964b39f954f7bd37307badc67de329a8a9341069 Nov 02 20:32:35.980106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.980075 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v"] Nov 02 20:32:35.982146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.982114 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:35.982211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.982176 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:35.982245 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:35.982213 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:36.430966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:36.430925 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" event={"ID":"7a943a64-646e-4533-a37c-71334481ec3a","Type":"ContainerStarted","Data":"199e900c640bece93cdb58672d49d1d38d0b04f0b5ae6c965d3f41e972f210f9"} Nov 02 20:32:36.431159 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:36.431000 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" event={"ID":"7a943a64-646e-4533-a37c-71334481ec3a","Type":"ContainerStarted","Data":"21c6277bb43c66fce9a1755c964b39f954f7bd37307badc67de329a8a9341069"} Nov 02 20:32:36.559634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:36.559588 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:36.598931 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:36.598877 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" podStartSLOduration=1.5988636440000001 podStartE2EDuration="1.598863644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:36.596428271 +0000 UTC m=+1084.886084798" watchObservedRunningTime="2026-11-02 20:32:36.598863644 +0000 UTC m=+1084.888520171" Nov 02 20:32:37.561796 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:37.561764 4971 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:32:37.562292 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:37.561820 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" podUID="7a943a64-646e-4533-a37c-71334481ec3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 20:32:37.861499 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:37.861467 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:38.559537 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:38.559502 4971 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:32:38.559727 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:38.559556 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" podUID="7a943a64-646e-4533-a37c-71334481ec3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 20:32:39.559939 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:39.559901 4971 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:32:39.560378 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:39.559960 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" podUID="7a943a64-646e-4533-a37c-71334481ec3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 20:32:40.560020 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:40.559986 4971 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:32:40.560417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:40.560038 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" podUID="7a943a64-646e-4533-a37c-71334481ec3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 20:32:41.560415 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:41.560378 4971 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:32:41.560815 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:41.560442 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" podUID="7a943a64-646e-4533-a37c-71334481ec3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 20:32:42.716572 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:42.716533 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:42.824375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:42.824324 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:42.824870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:42.824846 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:42.826128 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:42.826065 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:32:44.369211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.369156 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9"] Nov 02 20:32:44.376776 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.376752 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.426984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.426939 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-2250b4-openshift-default-dockercfg-qrvc5\"" Nov 02 20:32:44.481176 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.481146 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9"] Nov 02 20:32:44.482111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.482059 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.482417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.482394 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.482621 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.482588 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.482917 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.482897 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.483116 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.483074 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.483272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.483255 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.483423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.483404 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.483618 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.483585 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsghx\" (UniqueName: \"kubernetes.io/projected/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-kube-api-access-rsghx\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.483821 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.483804 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.585094 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.585058 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.585276 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.585109 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.585322 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.585294 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.585390 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.585367 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsghx\" (UniqueName: \"kubernetes.io/projected/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-kube-api-access-rsghx\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.585488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.585467 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.585593 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.585578 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.585660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.585606 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.585660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.585632 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.585845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.585717 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.586136 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.586066 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.586136 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.586117 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.586314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.586075 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.586378 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.586329 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.586671 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.586646 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.589444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.589413 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.589598 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.589579 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.647986 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.647849 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.648745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.648720 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsghx\" (UniqueName: \"kubernetes.io/projected/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-kube-api-access-rsghx\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:44.698006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:44.697982 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:45.124022 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:32:45.123991 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb590ab7b_774d_4cce_9cc4_c15ebbe5181b.slice/crio-2a72b284bafb94413cf700bf85dc10537972189f4bbbaa238bd846fc9f76cb7a WatchSource:0}: Error finding container 2a72b284bafb94413cf700bf85dc10537972189f4bbbaa238bd846fc9f76cb7a: Status 404 returned error can't find the container with id 2a72b284bafb94413cf700bf85dc10537972189f4bbbaa238bd846fc9f76cb7a Nov 02 20:32:45.124864 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:45.124835 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9"] Nov 02 20:32:45.126350 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:45.126318 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:45.126423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:45.126381 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:45.126423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:45.126409 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:45.511678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:45.511593 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" event={"ID":"b590ab7b-774d-4cce-9cc4-c15ebbe5181b","Type":"ContainerStarted","Data":"9a16382233ff47acdb32733847e899a5e125dce52a8c6d3bc94976e17b8c8544"} Nov 02 20:32:45.511678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:45.511642 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" event={"ID":"b590ab7b-774d-4cce-9cc4-c15ebbe5181b","Type":"ContainerStarted","Data":"2a72b284bafb94413cf700bf85dc10537972189f4bbbaa238bd846fc9f76cb7a"} Nov 02 20:32:45.645107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:45.643580 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" podStartSLOduration=1.643538712 podStartE2EDuration="1.643538712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:45.629428388 +0000 UTC m=+1093.919084917" watchObservedRunningTime="2026-11-02 20:32:45.643538712 +0000 UTC m=+1093.933195242" Nov 02 20:32:45.698770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:45.698740 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:46.700129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:46.700097 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 20:32:46.700557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:46.700156 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" podUID="b590ab7b-774d-4cce-9cc4-c15ebbe5181b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 20:32:47.549115 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:47.549034 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:47.698926 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:47.698891 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 20:32:47.699135 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:47.698944 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" podUID="b590ab7b-774d-4cce-9cc4-c15ebbe5181b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 20:32:48.698815 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:48.698780 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 20:32:48.699236 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:48.698843 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" podUID="b590ab7b-774d-4cce-9cc4-c15ebbe5181b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 20:32:49.415036 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.415003 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx"] Nov 02 20:32:49.419293 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.419278 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.469009 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.468977 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-30ee89-openshift-default-dockercfg-xj9qc\"" Nov 02 20:32:49.521947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.521917 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx"] Nov 02 20:32:49.568924 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.568883 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flr6f\" (UniqueName: \"kubernetes.io/projected/0d6924ca-50c1-496f-9bd0-fcaa96717981-kube-api-access-flr6f\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.569126 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.569062 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.569216 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.569125 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.569278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.569208 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.569336 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.569278 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6924ca-50c1-496f-9bd0-fcaa96717981-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.569416 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.569387 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.569477 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.569447 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.569534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.569473 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.569534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.569505 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.670615 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.670542 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.670615 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.670574 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.670615 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.670605 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.670853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.670629 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6924ca-50c1-496f-9bd0-fcaa96717981-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.670853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.670669 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.670853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.670741 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.670853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.670765 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.670853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.670791 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.670853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.670835 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flr6f\" (UniqueName: \"kubernetes.io/projected/0d6924ca-50c1-496f-9bd0-fcaa96717981-kube-api-access-flr6f\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.671173 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.671152 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.671285 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.671254 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.671417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.671380 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6924ca-50c1-496f-9bd0-fcaa96717981-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.671689 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.671672 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.671732 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.671672 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.673174 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.673154 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.673397 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.673378 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.699193 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.699172 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 20:32:49.699514 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.699222 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" podUID="b590ab7b-774d-4cce-9cc4-c15ebbe5181b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 20:32:49.749763 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.749736 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flr6f\" (UniqueName: \"kubernetes.io/projected/0d6924ca-50c1-496f-9bd0-fcaa96717981-kube-api-access-flr6f\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.750352 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.750329 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:49.757609 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:49.757585 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:50.182716 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:32:50.182688 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6924ca_50c1_496f_9bd0_fcaa96717981.slice/crio-ee8002fb4f56766ac94fe8745f4a94e95805be60ab1b9c588695f44a0259a8e6 WatchSource:0}: Error finding container ee8002fb4f56766ac94fe8745f4a94e95805be60ab1b9c588695f44a0259a8e6: Status 404 returned error can't find the container with id ee8002fb4f56766ac94fe8745f4a94e95805be60ab1b9c588695f44a0259a8e6 Nov 02 20:32:50.183646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:50.183618 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx"] Nov 02 20:32:50.185191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:50.185155 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:50.185283 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:50.185220 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:50.185283 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:50.185270 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:32:50.570820 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:50.570725 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" event={"ID":"0d6924ca-50c1-496f-9bd0-fcaa96717981","Type":"ContainerStarted","Data":"326a896a86532092e8b8e75ccc7b68d1b17bdc95e320c2911a9b8a99b1c96d3d"} Nov 02 20:32:50.570820 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:50.570788 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" event={"ID":"0d6924ca-50c1-496f-9bd0-fcaa96717981","Type":"ContainerStarted","Data":"ee8002fb4f56766ac94fe8745f4a94e95805be60ab1b9c588695f44a0259a8e6"} Nov 02 20:32:50.686996 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:50.686944 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" podStartSLOduration=1.6869290879999999 podStartE2EDuration="1.686929088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:50.684338121 +0000 UTC m=+1098.973994651" watchObservedRunningTime="2026-11-02 20:32:50.686929088 +0000 UTC m=+1098.976585616" Nov 02 20:32:50.699385 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:50.699354 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 20:32:50.699779 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:50.699405 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" podUID="b590ab7b-774d-4cce-9cc4-c15ebbe5181b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 20:32:50.759131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:50.758171 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:51.759817 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:51.759774 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 02 20:32:51.760294 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:51.759846 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" podUID="0d6924ca-50c1-496f-9bd0-fcaa96717981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 02 20:32:51.852355 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:51.852329 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:51.853155 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:51.853126 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:51.853710 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:51.853692 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:32:52.482573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:52.482535 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:52.758649 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:52.758578 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 02 20:32:52.758649 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:52.758632 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" podUID="0d6924ca-50c1-496f-9bd0-fcaa96717981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 02 20:32:53.758652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:53.758618 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 02 20:32:53.759146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:53.758686 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" podUID="0d6924ca-50c1-496f-9bd0-fcaa96717981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 02 20:32:54.758766 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:54.758736 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 02 20:32:54.759173 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:54.758785 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" podUID="0d6924ca-50c1-496f-9bd0-fcaa96717981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 02 20:32:55.758550 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:55.758514 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 02 20:32:55.760641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:55.758567 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" podUID="0d6924ca-50c1-496f-9bd0-fcaa96717981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 02 20:32:56.853442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:56.853409 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:56.854264 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:56.854237 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:56.855196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:56.855171 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:32:57.302456 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:57.302363 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:32:59.637509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:59.637478 4971 generic.go:356] "Generic (PLEG): container finished" podID="c485622a-8d91-4523-bd63-97cce75df918" containerID="664a1332fc9c6dc9776fc2c9906697adb1fa7dd812397e3222db4cb2a9335b0d" exitCode=0 Nov 02 20:32:59.637898 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:32:59.637550 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" event={"ID":"c485622a-8d91-4523-bd63-97cce75df918","Type":"ContainerDied","Data":"664a1332fc9c6dc9776fc2c9906697adb1fa7dd812397e3222db4cb2a9335b0d"} Nov 02 20:33:00.805423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:00.805399 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" Nov 02 20:33:00.927609 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:00.927537 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c485622a-8d91-4523-bd63-97cce75df918-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c485622a-8d91-4523-bd63-97cce75df918-trusted-ca-bundle\") pod \"c485622a-8d91-4523-bd63-97cce75df918\" (UID: \"c485622a-8d91-4523-bd63-97cce75df918\") " Nov 02 20:33:00.927609 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:00.927589 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c485622a-8d91-4523-bd63-97cce75df918-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c485622a-8d91-4523-bd63-97cce75df918-service-ca-bundle\") pod \"c485622a-8d91-4523-bd63-97cce75df918\" (UID: \"c485622a-8d91-4523-bd63-97cce75df918\") " Nov 02 20:33:00.927828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:00.927629 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c485622a-8d91-4523-bd63-97cce75df918-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c485622a-8d91-4523-bd63-97cce75df918-archives-path\") pod \"c485622a-8d91-4523-bd63-97cce75df918\" (UID: \"c485622a-8d91-4523-bd63-97cce75df918\") " Nov 02 20:33:00.927828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:00.927702 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c485622a-8d91-4523-bd63-97cce75df918-kube-api-access-wt2lf\" (UniqueName: \"kubernetes.io/projected/c485622a-8d91-4523-bd63-97cce75df918-kube-api-access-wt2lf\") pod \"c485622a-8d91-4523-bd63-97cce75df918\" (UID: \"c485622a-8d91-4523-bd63-97cce75df918\") " Nov 02 20:33:00.928025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:00.927998 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c485622a-8d91-4523-bd63-97cce75df918-service-ca-bundle" pod "c485622a-8d91-4523-bd63-97cce75df918" (UID: "c485622a-8d91-4523-bd63-97cce75df918"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:00.928025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:00.928010 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c485622a-8d91-4523-bd63-97cce75df918-trusted-ca-bundle" pod "c485622a-8d91-4523-bd63-97cce75df918" (UID: "c485622a-8d91-4523-bd63-97cce75df918"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:00.928187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:00.928065 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c485622a-8d91-4523-bd63-97cce75df918-archives-path" pod "c485622a-8d91-4523-bd63-97cce75df918" (UID: "c485622a-8d91-4523-bd63-97cce75df918"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:00.928411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:00.928389 4971 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c485622a-8d91-4523-bd63-97cce75df918-trusted-ca-bundle\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:00.928453 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:00.928418 4971 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c485622a-8d91-4523-bd63-97cce75df918-service-ca-bundle\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:00.928453 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:00.928433 4971 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c485622a-8d91-4523-bd63-97cce75df918-archives-path\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:00.930153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:00.930126 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c485622a-8d91-4523-bd63-97cce75df918-kube-api-access-wt2lf" pod "c485622a-8d91-4523-bd63-97cce75df918" (UID: "c485622a-8d91-4523-bd63-97cce75df918"). InnerVolumeSpecName "kube-api-access-wt2lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:01.029876 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:01.029662 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt2lf\" (UniqueName: \"kubernetes.io/projected/c485622a-8d91-4523-bd63-97cce75df918-kube-api-access-wt2lf\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:01.653535 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:01.653505 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" Nov 02 20:33:01.653702 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:01.653538 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gmvs9-mpwh2" event={"ID":"c485622a-8d91-4523-bd63-97cce75df918","Type":"ContainerDied","Data":"04901f86469c7c4f8e6df45e3a3b769015dc450985b59c11ecd96699fd41e6e3"} Nov 02 20:33:01.653702 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:01.653583 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04901f86469c7c4f8e6df45e3a3b769015dc450985b59c11ecd96699fd41e6e3" Nov 02 20:33:02.119896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:02.119859 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:02.878540 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:02.878508 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:03.127947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:03.127918 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:03.339191 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:33:03.339116 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:33:04.081216 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.081182 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:04.860272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.860226 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg"] Nov 02 20:33:04.861054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.861031 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c485622a-8d91-4523-bd63-97cce75df918" containerName="insights-gathering" Nov 02 20:33:04.861203 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.861055 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c485622a-8d91-4523-bd63-97cce75df918" containerName="insights-gathering" Nov 02 20:33:04.861267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.861252 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c485622a-8d91-4523-bd63-97cce75df918" containerName="insights-gathering" Nov 02 20:33:04.865153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.865133 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.885199 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.885172 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d27n\" (UniqueName: \"kubernetes.io/projected/697fdff1-eee0-4118-9af2-4b24a48359ea-kube-api-access-8d27n\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.885305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.885212 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.885305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.885233 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.885305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.885252 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.885415 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.885371 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.885452 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.885425 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/697fdff1-eee0-4118-9af2-4b24a48359ea-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.885492 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.885480 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.885534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.885520 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.885582 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.885571 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.971189 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.971153 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg"] Nov 02 20:33:04.986780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.986718 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d27n\" (UniqueName: \"kubernetes.io/projected/697fdff1-eee0-4118-9af2-4b24a48359ea-kube-api-access-8d27n\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.986780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.986778 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.987176 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.986811 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.987176 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.986842 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.987176 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.986889 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.987176 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.986963 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/697fdff1-eee0-4118-9af2-4b24a48359ea-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.987176 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.986994 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.987176 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.987058 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.987176 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.987147 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.988736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.988689 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.989320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.989283 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.989764 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.989743 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.990016 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.989995 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.990649 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.990625 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/697fdff1-eee0-4118-9af2-4b24a48359ea-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.994401 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.994377 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:04.995209 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:04.995182 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:05.046951 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:05.046929 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:05.047162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:05.047139 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d27n\" (UniqueName: \"kubernetes.io/projected/697fdff1-eee0-4118-9af2-4b24a48359ea-kube-api-access-8d27n\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:05.179362 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:05.179306 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:05.608990 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:33:05.608957 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697fdff1_eee0_4118_9af2_4b24a48359ea.slice/crio-a4fb3555ac8f739787c8faaa5304e2664caa9679c637746898d1bcff5cd896be WatchSource:0}: Error finding container a4fb3555ac8f739787c8faaa5304e2664caa9679c637746898d1bcff5cd896be: Status 404 returned error can't find the container with id a4fb3555ac8f739787c8faaa5304e2664caa9679c637746898d1bcff5cd896be Nov 02 20:33:05.610399 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:05.610368 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg"] Nov 02 20:33:05.611392 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:05.611360 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:33:05.611462 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:05.611424 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:33:05.611462 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:05.611453 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:33:05.684400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:05.684376 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" event={"ID":"697fdff1-eee0-4118-9af2-4b24a48359ea","Type":"ContainerStarted","Data":"60b388975fbc63c6da8f2c956800e1249ba245b9027643adc3ca7d6f0ade175e"} Nov 02 20:33:05.684504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:05.684426 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" event={"ID":"697fdff1-eee0-4118-9af2-4b24a48359ea","Type":"ContainerStarted","Data":"a4fb3555ac8f739787c8faaa5304e2664caa9679c637746898d1bcff5cd896be"} Nov 02 20:33:05.799876 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:05.799824 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" podStartSLOduration=1.799811487 podStartE2EDuration="1.799811487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:05.797613396 +0000 UTC m=+1114.087269924" watchObservedRunningTime="2026-11-02 20:33:05.799811487 +0000 UTC m=+1114.089468015" Nov 02 20:33:06.180217 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:06.180184 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:06.181633 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:06.181601 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 20:33:06.181771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:06.181662 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" podUID="697fdff1-eee0-4118-9af2-4b24a48359ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 20:33:06.946114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:06.945056 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:07.180015 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:07.179982 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 20:33:07.180231 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:07.180039 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" podUID="697fdff1-eee0-4118-9af2-4b24a48359ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 20:33:08.180128 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:08.180069 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 20:33:08.180576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:08.180548 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" podUID="697fdff1-eee0-4118-9af2-4b24a48359ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 20:33:08.347146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:08.337784 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:09.179592 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:09.179555 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 20:33:09.179747 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:09.179613 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" podUID="697fdff1-eee0-4118-9af2-4b24a48359ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 20:33:10.180366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:10.180317 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 20:33:10.180958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:10.180395 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" podUID="697fdff1-eee0-4118-9af2-4b24a48359ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 20:33:11.179785 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:11.179753 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 20:33:11.180115 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:11.180059 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" podUID="697fdff1-eee0-4118-9af2-4b24a48359ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 20:33:11.315161 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:11.315029 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:11.791283 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:11.791250 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:12.287143 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:12.287115 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:12.288019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:12.287993 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:12.289241 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:12.289214 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:33:14.647975 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:14.647939 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:16.612853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:16.612814 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:21.406693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:21.406658 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:25.816400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:25.816363 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:26.061382 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:26.061345 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:26.308111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:26.308051 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:28.038608 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:28.038576 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:31.006270 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:31.006239 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:33.366707 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:33:33.366665 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:33:35.902339 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:35.902302 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:36.694164 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.694133 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s"] Nov 02 20:33:36.699650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.699630 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.744966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.744933 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-model-cache\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.745170 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.745094 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bf9x\" (UniqueName: \"kubernetes.io/projected/0cfa705d-47bf-4d68-8f92-a9132c911b6b-kube-api-access-6bf9x\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.745268 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.745243 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-dshm\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.745332 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.745285 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-home\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.745421 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.745398 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cfa705d-47bf-4d68-8f92-a9132c911b6b-tls-certs\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.745540 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.745520 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-tmp-dir\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.756209 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.756109 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s"] Nov 02 20:33:36.798398 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.798377 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-61cdc1\"/\"distinct-w3-61cdc1-kserve-self-signed-certs\"" Nov 02 20:33:36.847139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.847105 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-dshm\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.847467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.847435 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-home\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.847685 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.847665 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cfa705d-47bf-4d68-8f92-a9132c911b6b-tls-certs\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.847848 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.847829 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-tmp-dir\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.848097 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.848057 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-model-cache\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.859353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.859303 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-home\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.860130 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.848438 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bf9x\" (UniqueName: \"kubernetes.io/projected/0cfa705d-47bf-4d68-8f92-a9132c911b6b-kube-api-access-6bf9x\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.866275 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.866248 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-dshm\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.868302 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.868273 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-tmp-dir\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.868721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.868701 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-model-cache\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.873485 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.873462 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cfa705d-47bf-4d68-8f92-a9132c911b6b-tls-certs\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:36.892389 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.892340 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:36.920788 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:36.920768 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bf9x\" (UniqueName: \"kubernetes.io/projected/0cfa705d-47bf-4d68-8f92-a9132c911b6b-kube-api-access-6bf9x\") pod \"distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:37.018062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:37.018000 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:37.428245 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:33:37.428205 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cfa705d_47bf_4d68_8f92_a9132c911b6b.slice/crio-8f2c662d1c73da4a7d6926b5dd097bb814987fb71c379c64143c228253a69787 WatchSource:0}: Error finding container 8f2c662d1c73da4a7d6926b5dd097bb814987fb71c379c64143c228253a69787: Status 404 returned error can't find the container with id 8f2c662d1c73da4a7d6926b5dd097bb814987fb71c379c64143c228253a69787 Nov 02 20:33:37.429237 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:37.429212 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s"] Nov 02 20:33:37.956336 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:37.956301 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" event={"ID":"0cfa705d-47bf-4d68-8f92-a9132c911b6b","Type":"ContainerStarted","Data":"1b44310cacce629cddf37549ff26e06391314064c0e9425a6f435f1b5b9e6d0d"} Nov 02 20:33:37.956716 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:37.956347 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" event={"ID":"0cfa705d-47bf-4d68-8f92-a9132c911b6b","Type":"ContainerStarted","Data":"8f2c662d1c73da4a7d6926b5dd097bb814987fb71c379c64143c228253a69787"} Nov 02 20:33:38.019041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:38.019012 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:38.067539 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:38.067493 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podStartSLOduration=2.067479226 podStartE2EDuration="2.067479226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:38.066231634 +0000 UTC m=+1146.355888165" watchObservedRunningTime="2026-11-02 20:33:38.067479226 +0000 UTC m=+1146.357135753" Nov 02 20:33:40.899170 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:40.899133 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:42.645267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:42.645234 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:45.743139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:45.743099 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:47.018942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:47.018912 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:47.033982 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:47.033949 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:47.050436 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:47.050412 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:33:50.569141 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:50.569108 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:55.439150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:55.439058 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:33:55.826397 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:33:55.826300 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:00.208801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:00.208765 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:03.339183 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:34:03.339150 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:34:05.175422 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:05.175390 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:10.183373 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:10.183322 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:15.116216 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:15.116173 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:20.017757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:20.017720 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:20.836030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:20.835997 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:24.810348 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:24.810308 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:28.313249 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:28.313214 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:29.611725 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:29.611686 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:32.006165 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:32.006134 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:34:32.006617 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:32.006587 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:34:33.338360 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:34:33.338325 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:34:34.410395 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:34.410366 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:35.869252 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:35.869225 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s"] Nov 02 20:34:35.874154 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:35.874132 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:35.936272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:35.936240 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s"] Nov 02 20:34:36.023771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.023739 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.023953 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.023791 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.023953 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.023808 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.023953 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.023903 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.024125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.023961 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9059c2-ef01-4073-9478-66013d3bf4cc-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.024125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.023986 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.024125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.024001 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.024125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.024047 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j8ht\" (UniqueName: \"kubernetes.io/projected/6f9059c2-ef01-4073-9478-66013d3bf4cc-kube-api-access-7j8ht\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.024301 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.024160 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.125721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.125624 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9059c2-ef01-4073-9478-66013d3bf4cc-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.125721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.125658 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.125721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.125676 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.125984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.125743 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j8ht\" (UniqueName: \"kubernetes.io/projected/6f9059c2-ef01-4073-9478-66013d3bf4cc-kube-api-access-7j8ht\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.125984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.125813 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.125984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.125892 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.125984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.125952 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.125984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.125976 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.126231 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.126010 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.126231 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.126115 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.126312 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.126261 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.126433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.126412 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.126515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.126489 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.126697 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.126681 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9059c2-ef01-4073-9478-66013d3bf4cc-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.128569 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.128542 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.128709 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.128672 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.183894 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.183868 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.184030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.184012 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j8ht\" (UniqueName: \"kubernetes.io/projected/6f9059c2-ef01-4073-9478-66013d3bf4cc-kube-api-access-7j8ht\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.188860 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.188844 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:36.615386 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.615357 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s"] Nov 02 20:34:36.646931 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:34:36.646899 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9059c2_ef01_4073_9478_66013d3bf4cc.slice/crio-59398373a90daff27a93a50ef612a89fb223cd394514b06af92661d04b769244 WatchSource:0}: Error finding container 59398373a90daff27a93a50ef612a89fb223cd394514b06af92661d04b769244: Status 404 returned error can't find the container with id 59398373a90daff27a93a50ef612a89fb223cd394514b06af92661d04b769244 Nov 02 20:34:36.649503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.649468 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:34:36.649606 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.649572 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:34:36.649672 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:36.649619 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:34:37.403098 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:37.403059 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" event={"ID":"6f9059c2-ef01-4073-9478-66013d3bf4cc","Type":"ContainerStarted","Data":"79bb69c75945b88876012b02266fdbe8fa747d610edfd1cdf3da81b80f61c982"} Nov 02 20:34:37.403511 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:37.403135 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" event={"ID":"6f9059c2-ef01-4073-9478-66013d3bf4cc","Type":"ContainerStarted","Data":"59398373a90daff27a93a50ef612a89fb223cd394514b06af92661d04b769244"} Nov 02 20:34:37.520176 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:37.520121 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" podStartSLOduration=2.520100992 podStartE2EDuration="2.520100992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:37.517638939 +0000 UTC m=+1205.807295468" watchObservedRunningTime="2026-11-02 20:34:37.520100992 +0000 UTC m=+1205.809757521" Nov 02 20:34:37.632681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:37.632646 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:38.094165 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:38.094127 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:38.189114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:38.189048 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:38.191099 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:38.191048 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 20:34:38.191261 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:38.191138 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" podUID="6f9059c2-ef01-4073-9478-66013d3bf4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 20:34:39.190054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:39.190016 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 20:34:39.190486 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:39.190068 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" podUID="6f9059c2-ef01-4073-9478-66013d3bf4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 20:34:39.355612 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:39.355577 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:40.189903 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:40.189867 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 20:34:40.190073 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:40.189937 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" podUID="6f9059c2-ef01-4073-9478-66013d3bf4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 20:34:41.189470 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:41.189426 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 20:34:41.189681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:41.189501 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" podUID="6f9059c2-ef01-4073-9478-66013d3bf4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 20:34:42.189693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:42.189654 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 20:34:42.190147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:42.189726 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" podUID="6f9059c2-ef01-4073-9478-66013d3bf4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 20:34:43.190178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:43.190148 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 02 20:34:43.190567 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:43.190219 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" podUID="6f9059c2-ef01-4073-9478-66013d3bf4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 02 20:34:44.128354 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:44.128315 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:44.194880 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:44.194852 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:44.195363 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:44.195343 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:44.196171 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:44.196153 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:34:48.942052 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:48.942014 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:53.739488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:53.739450 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:34:58.646640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:34:58.646602 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:00.685149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:00.685116 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp"] Nov 02 20:35:00.689282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:00.689263 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" Nov 02 20:35:00.743728 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:00.743707 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp"] Nov 02 20:35:00.781507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:00.781472 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rhhj\" (UniqueName: \"kubernetes.io/projected/38880071-38b8-459e-a7b2-9ab3875087bd-kube-api-access-7rhhj\") pod \"maas-api-revoke-sub-517db615-n7xzp\" (UID: \"38880071-38b8-459e-a7b2-9ab3875087bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" Nov 02 20:35:00.781636 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:00.781517 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38880071-38b8-459e-a7b2-9ab3875087bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-517db615-n7xzp\" (UID: \"38880071-38b8-459e-a7b2-9ab3875087bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" Nov 02 20:35:00.781711 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:00.781687 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38880071-38b8-459e-a7b2-9ab3875087bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-517db615-n7xzp\" (UID: \"38880071-38b8-459e-a7b2-9ab3875087bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" Nov 02 20:35:00.784663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:00.784647 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:35:00.784663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:00.784646 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:35:00.882494 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:00.882471 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38880071-38b8-459e-a7b2-9ab3875087bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-517db615-n7xzp\" (UID: \"38880071-38b8-459e-a7b2-9ab3875087bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" Nov 02 20:35:00.882597 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:00.882568 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rhhj\" (UniqueName: \"kubernetes.io/projected/38880071-38b8-459e-a7b2-9ab3875087bd-kube-api-access-7rhhj\") pod \"maas-api-revoke-sub-517db615-n7xzp\" (UID: \"38880071-38b8-459e-a7b2-9ab3875087bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" Nov 02 20:35:00.882643 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:00.882595 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38880071-38b8-459e-a7b2-9ab3875087bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-517db615-n7xzp\" (UID: \"38880071-38b8-459e-a7b2-9ab3875087bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" Nov 02 20:35:00.883056 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:00.883039 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38880071-38b8-459e-a7b2-9ab3875087bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-517db615-n7xzp\" (UID: \"38880071-38b8-459e-a7b2-9ab3875087bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" Nov 02 20:35:00.884954 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:00.884936 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38880071-38b8-459e-a7b2-9ab3875087bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-517db615-n7xzp\" (UID: \"38880071-38b8-459e-a7b2-9ab3875087bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" Nov 02 20:35:00.940397 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:00.940347 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rhhj\" (UniqueName: \"kubernetes.io/projected/38880071-38b8-459e-a7b2-9ab3875087bd-kube-api-access-7rhhj\") pod \"maas-api-revoke-sub-517db615-n7xzp\" (UID: \"38880071-38b8-459e-a7b2-9ab3875087bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" Nov 02 20:35:01.003343 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:01.003320 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" Nov 02 20:35:01.380814 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:35:01.380781 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38880071_38b8_459e_a7b2_9ab3875087bd.slice/crio-158eefc88b57ba5a2e2d78422c4028fa389fe5e429bb056af04cbaf4fe1eab7c WatchSource:0}: Error finding container 158eefc88b57ba5a2e2d78422c4028fa389fe5e429bb056af04cbaf4fe1eab7c: Status 404 returned error can't find the container with id 158eefc88b57ba5a2e2d78422c4028fa389fe5e429bb056af04cbaf4fe1eab7c Nov 02 20:35:01.381865 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:01.381842 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp"] Nov 02 20:35:01.644291 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:01.644262 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" event={"ID":"38880071-38b8-459e-a7b2-9ab3875087bd","Type":"ContainerStarted","Data":"91c4a62be57106d893f737d8aa75e98725237cf0bd1f6d0b7d3bce3a134dfbe6"} Nov 02 20:35:01.644413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:01.644320 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" event={"ID":"38880071-38b8-459e-a7b2-9ab3875087bd","Type":"ContainerStarted","Data":"158eefc88b57ba5a2e2d78422c4028fa389fe5e429bb056af04cbaf4fe1eab7c"} Nov 02 20:35:01.758027 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:01.757986 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" podStartSLOduration=1.757973743 podStartE2EDuration="1.757973743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:01.753494714 +0000 UTC m=+1230.043151243" watchObservedRunningTime="2026-11-02 20:35:01.757973743 +0000 UTC m=+1230.047630271" Nov 02 20:35:02.653499 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:02.653463 4971 generic.go:356] "Generic (PLEG): container finished" podID="38880071-38b8-459e-a7b2-9ab3875087bd" containerID="91c4a62be57106d893f737d8aa75e98725237cf0bd1f6d0b7d3bce3a134dfbe6" exitCode=0 Nov 02 20:35:02.653672 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:02.653547 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" event={"ID":"38880071-38b8-459e-a7b2-9ab3875087bd","Type":"ContainerDied","Data":"91c4a62be57106d893f737d8aa75e98725237cf0bd1f6d0b7d3bce3a134dfbe6"} Nov 02 20:35:03.340703 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:35:03.340666 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:35:03.528943 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:03.528832 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:03.825956 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:03.825933 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" Nov 02 20:35:03.917034 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:03.916971 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38880071-38b8-459e-a7b2-9ab3875087bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38880071-38b8-459e-a7b2-9ab3875087bd-maas-api-openshift-ca\") pod \"38880071-38b8-459e-a7b2-9ab3875087bd\" (UID: \"38880071-38b8-459e-a7b2-9ab3875087bd\") " Nov 02 20:35:03.917034 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:03.917032 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38880071-38b8-459e-a7b2-9ab3875087bd-kube-api-access-7rhhj\" (UniqueName: \"kubernetes.io/projected/38880071-38b8-459e-a7b2-9ab3875087bd-kube-api-access-7rhhj\") pod \"38880071-38b8-459e-a7b2-9ab3875087bd\" (UID: \"38880071-38b8-459e-a7b2-9ab3875087bd\") " Nov 02 20:35:03.917285 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:03.917106 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38880071-38b8-459e-a7b2-9ab3875087bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38880071-38b8-459e-a7b2-9ab3875087bd-maas-api-xks-ca\") pod \"38880071-38b8-459e-a7b2-9ab3875087bd\" (UID: \"38880071-38b8-459e-a7b2-9ab3875087bd\") " Nov 02 20:35:03.917348 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:03.917318 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38880071-38b8-459e-a7b2-9ab3875087bd-maas-api-openshift-ca" pod "38880071-38b8-459e-a7b2-9ab3875087bd" (UID: "38880071-38b8-459e-a7b2-9ab3875087bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:35:03.917618 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:03.917598 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38880071-38b8-459e-a7b2-9ab3875087bd-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:03.919985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:03.919958 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38880071-38b8-459e-a7b2-9ab3875087bd-kube-api-access-7rhhj" pod "38880071-38b8-459e-a7b2-9ab3875087bd" (UID: "38880071-38b8-459e-a7b2-9ab3875087bd"). InnerVolumeSpecName "kube-api-access-7rhhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:03.920069 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:03.919957 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38880071-38b8-459e-a7b2-9ab3875087bd-maas-api-xks-ca" pod "38880071-38b8-459e-a7b2-9ab3875087bd" (UID: "38880071-38b8-459e-a7b2-9ab3875087bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:35:04.019217 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:04.019188 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rhhj\" (UniqueName: \"kubernetes.io/projected/38880071-38b8-459e-a7b2-9ab3875087bd-kube-api-access-7rhhj\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:04.019217 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:04.019211 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38880071-38b8-459e-a7b2-9ab3875087bd-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:04.668924 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:04.668895 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" Nov 02 20:35:04.669351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:04.668940 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp" event={"ID":"38880071-38b8-459e-a7b2-9ab3875087bd","Type":"ContainerDied","Data":"158eefc88b57ba5a2e2d78422c4028fa389fe5e429bb056af04cbaf4fe1eab7c"} Nov 02 20:35:04.669351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:04.668987 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="158eefc88b57ba5a2e2d78422c4028fa389fe5e429bb056af04cbaf4fe1eab7c" Nov 02 20:35:08.307873 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:08.307839 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:09.420847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:09.420815 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:13.245378 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:13.245337 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:18.104293 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:18.104255 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:21.181071 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.181037 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z"] Nov 02 20:35:21.181673 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.181653 4971 cpu_manager.go:414] "containerMap: removing container" podUID="38880071-38b8-459e-a7b2-9ab3875087bd" containerName="revoke-keys" Nov 02 20:35:21.181673 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.181669 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38880071-38b8-459e-a7b2-9ab3875087bd" containerName="revoke-keys" Nov 02 20:35:21.181781 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.181758 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="38880071-38b8-459e-a7b2-9ab3875087bd" containerName="revoke-keys" Nov 02 20:35:21.185501 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.185481 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.245853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.245805 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z"] Nov 02 20:35:21.313947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.313917 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.314139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.313971 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.314139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.314096 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.314139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.314123 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ef03b8-5cfb-472d-9dff-31794d02143b-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.314352 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.314209 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.314352 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.314268 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhb2b\" (UniqueName: \"kubernetes.io/projected/01ef03b8-5cfb-472d-9dff-31794d02143b-kube-api-access-bhb2b\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.314444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.314350 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.314444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.314375 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.314521 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.314452 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.415519 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.415487 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.415648 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.415599 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.415690 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.415658 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.415748 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.415732 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.415882 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.415851 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ef03b8-5cfb-472d-9dff-31794d02143b-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.416039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.415902 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.416039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.415951 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhb2b\" (UniqueName: \"kubernetes.io/projected/01ef03b8-5cfb-472d-9dff-31794d02143b-kube-api-access-bhb2b\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.416039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.416009 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.416039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.416032 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.416294 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.416071 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.416294 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.416113 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.416566 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.416373 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.416566 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.416405 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.416566 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.416521 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ef03b8-5cfb-472d-9dff-31794d02143b-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.417948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.417926 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.418271 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.418252 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.476749 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.476694 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.476909 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.476887 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhb2b\" (UniqueName: \"kubernetes.io/projected/01ef03b8-5cfb-472d-9dff-31794d02143b-kube-api-access-bhb2b\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.499520 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.499499 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:21.929836 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:35:21.929807 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ef03b8_5cfb_472d_9dff_31794d02143b.slice/crio-1b427bbd588483171937362de909187065d3869432221d47f68f9e34b4d5a436 WatchSource:0}: Error finding container 1b427bbd588483171937362de909187065d3869432221d47f68f9e34b4d5a436: Status 404 returned error can't find the container with id 1b427bbd588483171937362de909187065d3869432221d47f68f9e34b4d5a436 Nov 02 20:35:21.930423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.930402 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z"] Nov 02 20:35:21.931937 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.931911 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:35:21.931998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.931969 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:35:21.932034 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:21.932000 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:35:22.799784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:22.799751 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" event={"ID":"01ef03b8-5cfb-472d-9dff-31794d02143b","Type":"ContainerStarted","Data":"e3395ae279d10874aee4dfdfb3d55cff2acc64ff9917a484604d1d28a47d3d29"} Nov 02 20:35:22.800180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:22.799805 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" event={"ID":"01ef03b8-5cfb-472d-9dff-31794d02143b","Type":"ContainerStarted","Data":"1b427bbd588483171937362de909187065d3869432221d47f68f9e34b4d5a436"} Nov 02 20:35:22.962726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:22.962672 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" podStartSLOduration=1.962659151 podStartE2EDuration="1.962659151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:22.960585059 +0000 UTC m=+1251.250241584" watchObservedRunningTime="2026-11-02 20:35:22.962659151 +0000 UTC m=+1251.252315680" Nov 02 20:35:23.062317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:23.062211 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:23.499701 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:23.499665 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:23.501235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:23.501207 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:35:23.501365 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:23.501262 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" podUID="01ef03b8-5cfb-472d-9dff-31794d02143b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 20:35:24.500817 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:24.500786 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:35:24.501309 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:24.500842 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" podUID="01ef03b8-5cfb-472d-9dff-31794d02143b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 20:35:25.500464 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:25.500430 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:35:25.500612 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:25.500481 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" podUID="01ef03b8-5cfb-472d-9dff-31794d02143b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 20:35:26.499980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:26.499945 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:35:26.500381 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:26.499993 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" podUID="01ef03b8-5cfb-472d-9dff-31794d02143b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 20:35:27.500357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.500323 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:35:27.500780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.500374 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" podUID="01ef03b8-5cfb-472d-9dff-31794d02143b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 20:35:27.550941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.550908 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s"] Nov 02 20:35:27.555300 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.555277 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" Nov 02 20:35:27.600891 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.600863 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s"] Nov 02 20:35:27.604420 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.604399 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:35:27.608650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.608625 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db55j\" (UniqueName: \"kubernetes.io/projected/656598b3-b015-49b7-b955-55cd54861736-kube-api-access-db55j\") pod \"maas-api-revoke-sub-67b9a426-mw79s\" (UID: \"656598b3-b015-49b7-b955-55cd54861736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" Nov 02 20:35:27.608746 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.608690 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656598b3-b015-49b7-b955-55cd54861736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67b9a426-mw79s\" (UID: \"656598b3-b015-49b7-b955-55cd54861736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" Nov 02 20:35:27.608746 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.608737 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656598b3-b015-49b7-b955-55cd54861736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67b9a426-mw79s\" (UID: \"656598b3-b015-49b7-b955-55cd54861736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" Nov 02 20:35:27.647501 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.647452 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:35:27.709543 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.709515 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db55j\" (UniqueName: \"kubernetes.io/projected/656598b3-b015-49b7-b955-55cd54861736-kube-api-access-db55j\") pod \"maas-api-revoke-sub-67b9a426-mw79s\" (UID: \"656598b3-b015-49b7-b955-55cd54861736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" Nov 02 20:35:27.709674 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.709581 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656598b3-b015-49b7-b955-55cd54861736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67b9a426-mw79s\" (UID: \"656598b3-b015-49b7-b955-55cd54861736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" Nov 02 20:35:27.709674 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.709621 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656598b3-b015-49b7-b955-55cd54861736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67b9a426-mw79s\" (UID: \"656598b3-b015-49b7-b955-55cd54861736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" Nov 02 20:35:27.710255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.710235 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656598b3-b015-49b7-b955-55cd54861736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67b9a426-mw79s\" (UID: \"656598b3-b015-49b7-b955-55cd54861736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" Nov 02 20:35:27.712099 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.712065 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656598b3-b015-49b7-b955-55cd54861736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67b9a426-mw79s\" (UID: \"656598b3-b015-49b7-b955-55cd54861736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" Nov 02 20:35:27.767519 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.767458 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db55j\" (UniqueName: \"kubernetes.io/projected/656598b3-b015-49b7-b955-55cd54861736-kube-api-access-db55j\") pod \"maas-api-revoke-sub-67b9a426-mw79s\" (UID: \"656598b3-b015-49b7-b955-55cd54861736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" Nov 02 20:35:27.869432 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:27.869404 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" Nov 02 20:35:28.258828 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:35:28.258775 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656598b3_b015_49b7_b955_55cd54861736.slice/crio-19b87ff550212aa23ed356e5ec257a0b435ca42ead34934a592c5a2935d02ef5 WatchSource:0}: Error finding container 19b87ff550212aa23ed356e5ec257a0b435ca42ead34934a592c5a2935d02ef5: Status 404 returned error can't find the container with id 19b87ff550212aa23ed356e5ec257a0b435ca42ead34934a592c5a2935d02ef5 Nov 02 20:35:28.259932 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:28.259907 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s"] Nov 02 20:35:28.419465 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:28.419424 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:28.500250 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:28.500223 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:35:28.500400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:28.500279 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" podUID="01ef03b8-5cfb-472d-9dff-31794d02143b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 20:35:28.735360 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:28.735329 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:28.867487 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:28.867454 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" event={"ID":"656598b3-b015-49b7-b955-55cd54861736","Type":"ContainerStarted","Data":"26ca336cd2d192736ffb46188c94d7f294f064e3a34d3888b14f809a1bef5e50"} Nov 02 20:35:28.867667 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:28.867552 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" event={"ID":"656598b3-b015-49b7-b955-55cd54861736","Type":"ContainerStarted","Data":"19b87ff550212aa23ed356e5ec257a0b435ca42ead34934a592c5a2935d02ef5"} Nov 02 20:35:28.979627 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:28.979576 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" podStartSLOduration=1.979562024 podStartE2EDuration="1.979562024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:28.977475265 +0000 UTC m=+1257.267131794" watchObservedRunningTime="2026-11-02 20:35:28.979562024 +0000 UTC m=+1257.269218550" Nov 02 20:35:29.505684 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:29.505648 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:29.506226 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:29.506206 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:29.507105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:29.507064 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:35:29.877194 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:29.877156 4971 generic.go:356] "Generic (PLEG): container finished" podID="656598b3-b015-49b7-b955-55cd54861736" containerID="26ca336cd2d192736ffb46188c94d7f294f064e3a34d3888b14f809a1bef5e50" exitCode=0 Nov 02 20:35:29.877194 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:29.877235 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" event={"ID":"656598b3-b015-49b7-b955-55cd54861736","Type":"ContainerDied","Data":"26ca336cd2d192736ffb46188c94d7f294f064e3a34d3888b14f809a1bef5e50"} Nov 02 20:35:31.074135 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:31.074074 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" Nov 02 20:35:31.195810 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:31.195736 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/656598b3-b015-49b7-b955-55cd54861736-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656598b3-b015-49b7-b955-55cd54861736-maas-api-xks-ca\") pod \"656598b3-b015-49b7-b955-55cd54861736\" (UID: \"656598b3-b015-49b7-b955-55cd54861736\") " Nov 02 20:35:31.195965 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:31.195921 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/656598b3-b015-49b7-b955-55cd54861736-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656598b3-b015-49b7-b955-55cd54861736-maas-api-openshift-ca\") pod \"656598b3-b015-49b7-b955-55cd54861736\" (UID: \"656598b3-b015-49b7-b955-55cd54861736\") " Nov 02 20:35:31.195965 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:31.195951 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/656598b3-b015-49b7-b955-55cd54861736-kube-api-access-db55j\" (UniqueName: \"kubernetes.io/projected/656598b3-b015-49b7-b955-55cd54861736-kube-api-access-db55j\") pod \"656598b3-b015-49b7-b955-55cd54861736\" (UID: \"656598b3-b015-49b7-b955-55cd54861736\") " Nov 02 20:35:31.196320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:31.196285 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/656598b3-b015-49b7-b955-55cd54861736-maas-api-openshift-ca" pod "656598b3-b015-49b7-b955-55cd54861736" (UID: "656598b3-b015-49b7-b955-55cd54861736"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:35:31.196669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:31.196640 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656598b3-b015-49b7-b955-55cd54861736-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:31.198075 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:31.198047 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/656598b3-b015-49b7-b955-55cd54861736-maas-api-xks-ca" pod "656598b3-b015-49b7-b955-55cd54861736" (UID: "656598b3-b015-49b7-b955-55cd54861736"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:35:31.198774 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:31.198757 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/656598b3-b015-49b7-b955-55cd54861736-kube-api-access-db55j" pod "656598b3-b015-49b7-b955-55cd54861736" (UID: "656598b3-b015-49b7-b955-55cd54861736"). InnerVolumeSpecName "kube-api-access-db55j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:31.297827 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:31.297803 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656598b3-b015-49b7-b955-55cd54861736-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:31.297827 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:31.297833 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db55j\" (UniqueName: \"kubernetes.io/projected/656598b3-b015-49b7-b955-55cd54861736-kube-api-access-db55j\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:31.895792 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:31.895758 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" event={"ID":"656598b3-b015-49b7-b955-55cd54861736","Type":"ContainerDied","Data":"19b87ff550212aa23ed356e5ec257a0b435ca42ead34934a592c5a2935d02ef5"} Nov 02 20:35:31.895982 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:31.895768 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s" Nov 02 20:35:31.895982 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:31.895814 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19b87ff550212aa23ed356e5ec257a0b435ca42ead34934a592c5a2935d02ef5" Nov 02 20:35:33.302019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:33.301992 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:33.339971 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:35:33.339942 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:35:37.877301 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:37.877270 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s"] Nov 02 20:35:37.877863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:37.877849 4971 cpu_manager.go:414] "containerMap: removing container" podUID="656598b3-b015-49b7-b955-55cd54861736" containerName="revoke-keys" Nov 02 20:35:37.877863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:37.877863 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="656598b3-b015-49b7-b955-55cd54861736" containerName="revoke-keys" Nov 02 20:35:37.877983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:37.877974 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="656598b3-b015-49b7-b955-55cd54861736" containerName="revoke-keys" Nov 02 20:35:37.882730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:37.882713 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:37.935888 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:37.935860 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s"] Nov 02 20:35:37.970218 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:37.970182 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-model-cache\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:37.970370 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:37.970269 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e44495a-efac-44a6-9077-724e39fa4d6f-tls-certs\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:37.970418 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:37.970396 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-dshm\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:37.970543 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:37.970527 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-home\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:37.970592 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:37.970556 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-tmp-dir\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:37.970706 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:37.970687 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c5sh\" (UniqueName: \"kubernetes.io/projected/4e44495a-efac-44a6-9077-724e39fa4d6f-kube-api-access-8c5sh\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:37.977343 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:37.977322 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-30ee89\"/\"distinct-2-w0-30ee89-kserve-self-signed-certs\"" Nov 02 20:35:38.066436 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.066403 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:38.072165 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.072125 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-home\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:38.073277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.072177 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-tmp-dir\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:38.073277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.072288 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c5sh\" (UniqueName: \"kubernetes.io/projected/4e44495a-efac-44a6-9077-724e39fa4d6f-kube-api-access-8c5sh\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:38.073277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.072413 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-model-cache\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:38.073277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.072466 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e44495a-efac-44a6-9077-724e39fa4d6f-tls-certs\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:38.073277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.072523 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-dshm\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:38.073277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.072622 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-home\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:38.073277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.073202 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-tmp-dir\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:38.073973 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.073459 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-model-cache\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:38.078163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.076035 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e44495a-efac-44a6-9077-724e39fa4d6f-tls-certs\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:38.078163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.076161 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-dshm\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:38.130785 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.130717 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c5sh\" (UniqueName: \"kubernetes.io/projected/4e44495a-efac-44a6-9077-724e39fa4d6f-kube-api-access-8c5sh\") pod \"distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:38.195984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.195958 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:38.328024 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.327910 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:38.611315 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:35:38.611285 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e44495a_efac_44a6_9077_724e39fa4d6f.slice/crio-9408f144f927c31d0aa7c166f696647edc995cc84c1746f6a6dd0ef2feea0337 WatchSource:0}: Error finding container 9408f144f927c31d0aa7c166f696647edc995cc84c1746f6a6dd0ef2feea0337: Status 404 returned error can't find the container with id 9408f144f927c31d0aa7c166f696647edc995cc84c1746f6a6dd0ef2feea0337 Nov 02 20:35:38.612786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.612757 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s"] Nov 02 20:35:38.949707 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.949631 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" event={"ID":"4e44495a-efac-44a6-9077-724e39fa4d6f","Type":"ContainerStarted","Data":"6c335aac4e8ab3d8c80453327d200e9734b824b441b0f9ab1eaef85540a20e30"} Nov 02 20:35:38.949707 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:38.949671 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" event={"ID":"4e44495a-efac-44a6-9077-724e39fa4d6f","Type":"ContainerStarted","Data":"9408f144f927c31d0aa7c166f696647edc995cc84c1746f6a6dd0ef2feea0337"} Nov 02 20:35:39.064107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:39.064036 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podStartSLOduration=2.064023088 podStartE2EDuration="2.064023088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:39.059541597 +0000 UTC m=+1267.349198125" watchObservedRunningTime="2026-11-02 20:35:39.064023088 +0000 UTC m=+1267.353679623" Nov 02 20:35:39.196390 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:39.196356 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:40.839565 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:40.839528 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n"] Nov 02 20:35:40.844113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:40.844073 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:40.893589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:40.893561 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-2250b4\"/\"openshift-service-ca.crt\"" Nov 02 20:35:40.899380 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:40.899354 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n"] Nov 02 20:35:40.939507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:40.939479 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-2250b4\"/\"kube-root-ca.crt\"" Nov 02 20:35:40.939632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:40.939598 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2250b4\"/\"default-dockercfg-swxfm\"" Nov 02 20:35:40.939632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:40.939600 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2250b4\"/\"distinct-2-w1-2250b4-kserve-self-signed-certs\"" Nov 02 20:35:41.008447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.008411 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhgkg\" (UniqueName: \"kubernetes.io/projected/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-kube-api-access-fhgkg\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.008626 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.008487 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-home\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.008626 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.008602 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-model-cache\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.008801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.008779 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-tls-certs\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.008841 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.008815 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-tmp-dir\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.008944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.008920 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-dshm\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.036384 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.036354 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:41.109849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.109820 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhgkg\" (UniqueName: \"kubernetes.io/projected/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-kube-api-access-fhgkg\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.110005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.109891 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-home\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.110005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.109917 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-model-cache\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.110005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.109996 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-tls-certs\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.110187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.110015 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-tmp-dir\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.110187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.110112 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-dshm\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.110390 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.110366 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-home\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.110461 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.110415 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-model-cache\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.110580 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.110547 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-tmp-dir\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.112588 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.112566 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-dshm\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.112849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.112831 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-tls-certs\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.167608 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.167583 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhgkg\" (UniqueName: \"kubernetes.io/projected/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-kube-api-access-fhgkg\") pod \"distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.457897 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.457818 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:41.630531 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.630493 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n"] Nov 02 20:35:41.636004 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.635983 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" Nov 02 20:35:41.685467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.685426 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:35:41.685601 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.685586 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:35:41.689142 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.689115 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n"] Nov 02 20:35:41.819370 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.819303 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfrp6\" (UniqueName: \"kubernetes.io/projected/ff22da18-e34e-4735-ac05-daae7268fb06-kube-api-access-rfrp6\") pod \"maas-api-revoke-sub-e3ea48cd-j8h4n\" (UID: \"ff22da18-e34e-4735-ac05-daae7268fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" Nov 02 20:35:41.819490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.819376 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff22da18-e34e-4735-ac05-daae7268fb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3ea48cd-j8h4n\" (UID: \"ff22da18-e34e-4735-ac05-daae7268fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" Nov 02 20:35:41.819536 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.819480 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff22da18-e34e-4735-ac05-daae7268fb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3ea48cd-j8h4n\" (UID: \"ff22da18-e34e-4735-ac05-daae7268fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" Nov 02 20:35:41.893230 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:35:41.893202 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7fe7734_b7d6_497e_818a_aa48fe34d4c9.slice/crio-3193c35d4b8e8c7ff99daaf0ebe525c4df4a73bacf8f485f2346a3dac6ad8f1a WatchSource:0}: Error finding container 3193c35d4b8e8c7ff99daaf0ebe525c4df4a73bacf8f485f2346a3dac6ad8f1a: Status 404 returned error can't find the container with id 3193c35d4b8e8c7ff99daaf0ebe525c4df4a73bacf8f485f2346a3dac6ad8f1a Nov 02 20:35:41.894377 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.894353 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n"] Nov 02 20:35:41.921069 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.921046 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrp6\" (UniqueName: \"kubernetes.io/projected/ff22da18-e34e-4735-ac05-daae7268fb06-kube-api-access-rfrp6\") pod \"maas-api-revoke-sub-e3ea48cd-j8h4n\" (UID: \"ff22da18-e34e-4735-ac05-daae7268fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" Nov 02 20:35:41.921194 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.921133 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff22da18-e34e-4735-ac05-daae7268fb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3ea48cd-j8h4n\" (UID: \"ff22da18-e34e-4735-ac05-daae7268fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" Nov 02 20:35:41.921194 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.921181 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff22da18-e34e-4735-ac05-daae7268fb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3ea48cd-j8h4n\" (UID: \"ff22da18-e34e-4735-ac05-daae7268fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" Nov 02 20:35:41.921714 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.921688 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff22da18-e34e-4735-ac05-daae7268fb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3ea48cd-j8h4n\" (UID: \"ff22da18-e34e-4735-ac05-daae7268fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" Nov 02 20:35:41.923473 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.923450 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff22da18-e34e-4735-ac05-daae7268fb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3ea48cd-j8h4n\" (UID: \"ff22da18-e34e-4735-ac05-daae7268fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" Nov 02 20:35:41.974775 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.974739 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" event={"ID":"b7fe7734-b7d6-497e-818a-aa48fe34d4c9","Type":"ContainerStarted","Data":"11d3d4c4d4779e4870a068882da070abaa134ad00515be3b05ec631b3dffab9a"} Nov 02 20:35:41.974882 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.974788 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" event={"ID":"b7fe7734-b7d6-497e-818a-aa48fe34d4c9","Type":"ContainerStarted","Data":"3193c35d4b8e8c7ff99daaf0ebe525c4df4a73bacf8f485f2346a3dac6ad8f1a"} Nov 02 20:35:41.983265 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:41.983235 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfrp6\" (UniqueName: \"kubernetes.io/projected/ff22da18-e34e-4735-ac05-daae7268fb06-kube-api-access-rfrp6\") pod \"maas-api-revoke-sub-e3ea48cd-j8h4n\" (UID: \"ff22da18-e34e-4735-ac05-daae7268fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" Nov 02 20:35:42.086942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:42.086856 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podStartSLOduration=2.086844827 podStartE2EDuration="2.086844827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:42.084237353 +0000 UTC m=+1270.373893880" watchObservedRunningTime="2026-11-02 20:35:42.086844827 +0000 UTC m=+1270.376501354" Nov 02 20:35:42.250430 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:42.250394 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" Nov 02 20:35:42.461458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:42.459365 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:42.623338 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:35:42.623306 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff22da18_e34e_4735_ac05_daae7268fb06.slice/crio-b78b51209be08ec5f65f4d0e9b82b4bc10521603b150798fc91848b2f12b9cc7 WatchSource:0}: Error finding container b78b51209be08ec5f65f4d0e9b82b4bc10521603b150798fc91848b2f12b9cc7: Status 404 returned error can't find the container with id b78b51209be08ec5f65f4d0e9b82b4bc10521603b150798fc91848b2f12b9cc7 Nov 02 20:35:42.623633 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:42.623608 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n"] Nov 02 20:35:42.984283 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:42.984250 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" event={"ID":"ff22da18-e34e-4735-ac05-daae7268fb06","Type":"ContainerStarted","Data":"fb0b2b9c8681456aa8912f30e008b7a0e9dff9a3c72311cc1fbbca62ee45d61e"} Nov 02 20:35:42.984616 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:42.984314 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" event={"ID":"ff22da18-e34e-4735-ac05-daae7268fb06","Type":"ContainerStarted","Data":"b78b51209be08ec5f65f4d0e9b82b4bc10521603b150798fc91848b2f12b9cc7"} Nov 02 20:35:43.096069 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:43.095928 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" podStartSLOduration=2.095910018 podStartE2EDuration="2.095910018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:43.095698348 +0000 UTC m=+1271.385354874" watchObservedRunningTime="2026-11-02 20:35:43.095910018 +0000 UTC m=+1271.385566547" Nov 02 20:35:43.101416 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:43.101383 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:43.992859 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:43.992816 4971 generic.go:356] "Generic (PLEG): container finished" podID="ff22da18-e34e-4735-ac05-daae7268fb06" containerID="fb0b2b9c8681456aa8912f30e008b7a0e9dff9a3c72311cc1fbbca62ee45d61e" exitCode=0 Nov 02 20:35:43.993337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:43.992900 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" event={"ID":"ff22da18-e34e-4735-ac05-daae7268fb06","Type":"ContainerDied","Data":"fb0b2b9c8681456aa8912f30e008b7a0e9dff9a3c72311cc1fbbca62ee45d61e"} Nov 02 20:35:45.176605 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:45.176583 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" Nov 02 20:35:45.259594 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:45.259534 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff22da18-e34e-4735-ac05-daae7268fb06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff22da18-e34e-4735-ac05-daae7268fb06-maas-api-xks-ca\") pod \"ff22da18-e34e-4735-ac05-daae7268fb06\" (UID: \"ff22da18-e34e-4735-ac05-daae7268fb06\") " Nov 02 20:35:45.259721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:45.259675 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff22da18-e34e-4735-ac05-daae7268fb06-kube-api-access-rfrp6\" (UniqueName: \"kubernetes.io/projected/ff22da18-e34e-4735-ac05-daae7268fb06-kube-api-access-rfrp6\") pod \"ff22da18-e34e-4735-ac05-daae7268fb06\" (UID: \"ff22da18-e34e-4735-ac05-daae7268fb06\") " Nov 02 20:35:45.259790 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:45.259726 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff22da18-e34e-4735-ac05-daae7268fb06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff22da18-e34e-4735-ac05-daae7268fb06-maas-api-openshift-ca\") pod \"ff22da18-e34e-4735-ac05-daae7268fb06\" (UID: \"ff22da18-e34e-4735-ac05-daae7268fb06\") " Nov 02 20:35:45.260099 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:45.260064 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff22da18-e34e-4735-ac05-daae7268fb06-maas-api-openshift-ca" pod "ff22da18-e34e-4735-ac05-daae7268fb06" (UID: "ff22da18-e34e-4735-ac05-daae7268fb06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:35:45.260214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:45.260198 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff22da18-e34e-4735-ac05-daae7268fb06-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:45.261669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:45.261651 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff22da18-e34e-4735-ac05-daae7268fb06-maas-api-xks-ca" pod "ff22da18-e34e-4735-ac05-daae7268fb06" (UID: "ff22da18-e34e-4735-ac05-daae7268fb06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:35:45.261795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:45.261781 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff22da18-e34e-4735-ac05-daae7268fb06-kube-api-access-rfrp6" pod "ff22da18-e34e-4735-ac05-daae7268fb06" (UID: "ff22da18-e34e-4735-ac05-daae7268fb06"). InnerVolumeSpecName "kube-api-access-rfrp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:45.360955 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:45.360924 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff22da18-e34e-4735-ac05-daae7268fb06-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:45.360955 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:45.360951 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfrp6\" (UniqueName: \"kubernetes.io/projected/ff22da18-e34e-4735-ac05-daae7268fb06-kube-api-access-rfrp6\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:46.012403 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:46.012368 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" Nov 02 20:35:46.012403 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:46.012393 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n" event={"ID":"ff22da18-e34e-4735-ac05-daae7268fb06","Type":"ContainerDied","Data":"b78b51209be08ec5f65f4d0e9b82b4bc10521603b150798fc91848b2f12b9cc7"} Nov 02 20:35:46.012618 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:46.012441 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b78b51209be08ec5f65f4d0e9b82b4bc10521603b150798fc91848b2f12b9cc7" Nov 02 20:35:46.150587 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:46.150489 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:48.015880 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:48.015837 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:48.211146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:48.201652 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:48.229938 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:48.229908 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:49.317425 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:49.317394 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:35:51.458123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:51.458065 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:51.474868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:51.474839 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:52.170713 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.170683 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:35:52.396508 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.396461 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr"] Nov 02 20:35:52.397263 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.397242 4971 cpu_manager.go:414] "containerMap: removing container" podUID="ff22da18-e34e-4735-ac05-daae7268fb06" containerName="revoke-keys" Nov 02 20:35:52.397263 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.397261 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff22da18-e34e-4735-ac05-daae7268fb06" containerName="revoke-keys" Nov 02 20:35:52.397427 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.397418 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff22da18-e34e-4735-ac05-daae7268fb06" containerName="revoke-keys" Nov 02 20:35:52.401204 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.401186 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr" Nov 02 20:35:52.450685 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.450602 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:35:52.450872 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.450800 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:35:52.454510 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.454489 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr"] Nov 02 20:35:52.501998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.501968 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de529379-dlznr\" (UID: \"fd8af1ff-e779-4317-b2f3-e9372d6e3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr" Nov 02 20:35:52.502398 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.502020 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsdnm\" (UniqueName: \"kubernetes.io/projected/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-kube-api-access-zsdnm\") pod \"maas-api-revoke-sub-de529379-dlznr\" (UID: \"fd8af1ff-e779-4317-b2f3-e9372d6e3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr" Nov 02 20:35:52.502398 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.502204 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de529379-dlznr\" (UID: \"fd8af1ff-e779-4317-b2f3-e9372d6e3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr" Nov 02 20:35:52.603743 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.603716 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de529379-dlznr\" (UID: \"fd8af1ff-e779-4317-b2f3-e9372d6e3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr" Nov 02 20:35:52.603847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.603754 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsdnm\" (UniqueName: \"kubernetes.io/projected/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-kube-api-access-zsdnm\") pod \"maas-api-revoke-sub-de529379-dlznr\" (UID: \"fd8af1ff-e779-4317-b2f3-e9372d6e3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr" Nov 02 20:35:52.603847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.603796 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de529379-dlznr\" (UID: \"fd8af1ff-e779-4317-b2f3-e9372d6e3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr" Nov 02 20:35:52.604439 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.604417 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de529379-dlznr\" (UID: \"fd8af1ff-e779-4317-b2f3-e9372d6e3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr" Nov 02 20:35:52.606397 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.606370 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de529379-dlznr\" (UID: \"fd8af1ff-e779-4317-b2f3-e9372d6e3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr" Nov 02 20:35:52.661564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.661545 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsdnm\" (UniqueName: \"kubernetes.io/projected/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-kube-api-access-zsdnm\") pod \"maas-api-revoke-sub-de529379-dlznr\" (UID: \"fd8af1ff-e779-4317-b2f3-e9372d6e3a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr" Nov 02 20:35:52.715230 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.715180 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr" Nov 02 20:35:52.947900 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:52.947848 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:53.089910 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:35:53.089880 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8af1ff_e779_4317_b2f3_e9372d6e3a31.slice/crio-5335df24fb988850224ff8e045d29d7ab1a046ea65871b004183b3d4094c2d94 WatchSource:0}: Error finding container 5335df24fb988850224ff8e045d29d7ab1a046ea65871b004183b3d4094c2d94: Status 404 returned error can't find the container with id 5335df24fb988850224ff8e045d29d7ab1a046ea65871b004183b3d4094c2d94 Nov 02 20:35:53.091041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.091013 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr"] Nov 02 20:35:53.169323 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.169297 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr" event={"ID":"fd8af1ff-e779-4317-b2f3-e9372d6e3a31","Type":"ContainerStarted","Data":"5335df24fb988850224ff8e045d29d7ab1a046ea65871b004183b3d4094c2d94"} Nov 02 20:35:53.532802 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.532774 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7"] Nov 02 20:35:53.537052 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.537037 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.591980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.591900 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7"] Nov 02 20:35:53.614808 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.614782 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-dshm\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.614969 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.614828 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-model-cache\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.614969 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.614855 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c20ca15-e559-4eb5-a92c-c0049a6e0308-tls-certs\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.615116 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.615005 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-home\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.615116 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.615043 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-tmp-dir\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.615235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.615204 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnnpk\" (UniqueName: \"kubernetes.io/projected/5c20ca15-e559-4eb5-a92c-c0049a6e0308-kube-api-access-mnnpk\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.632843 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.632820 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-694cee\"/\"distinct-2-w2-694cee-kserve-self-signed-certs\"" Nov 02 20:35:53.716222 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.716184 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnnpk\" (UniqueName: \"kubernetes.io/projected/5c20ca15-e559-4eb5-a92c-c0049a6e0308-kube-api-access-mnnpk\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.716542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.716406 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-dshm\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.716542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.716464 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-model-cache\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.716542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.716493 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c20ca15-e559-4eb5-a92c-c0049a6e0308-tls-certs\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.716987 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.716551 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-home\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.716987 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.716592 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-tmp-dir\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.717245 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.717220 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-model-cache\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.717346 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.717257 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-tmp-dir\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.717440 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.717418 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-home\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.720895 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.720829 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-dshm\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.723919 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.723045 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c20ca15-e559-4eb5-a92c-c0049a6e0308-tls-certs\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.724860 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.724834 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:35:53.773971 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.773942 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnnpk\" (UniqueName: \"kubernetes.io/projected/5c20ca15-e559-4eb5-a92c-c0049a6e0308-kube-api-access-mnnpk\") pod \"distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:53.849695 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:53.849666 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:54.178668 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:54.178597 4971 generic.go:356] "Generic (PLEG): container finished" podID="fd8af1ff-e779-4317-b2f3-e9372d6e3a31" containerID="ef21fd106740179d2952005d98f5a043d0723d8290e0eb6289bbb673c893fe0f" exitCode=0 Nov 02 20:35:54.178802 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:54.178684 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr" event={"ID":"fd8af1ff-e779-4317-b2f3-e9372d6e3a31","Type":"ContainerDied","Data":"ef21fd106740179d2952005d98f5a043d0723d8290e0eb6289bbb673c893fe0f"} Nov 02 20:35:54.243067 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:35:54.243042 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c20ca15_e559_4eb5_a92c_c0049a6e0308.slice/crio-b411e306a8126a3dbd841abd62af93954f191dd95e5ea53cd975a77d57bfb8c1 WatchSource:0}: Error finding container b411e306a8126a3dbd841abd62af93954f191dd95e5ea53cd975a77d57bfb8c1: Status 404 returned error can't find the container with id b411e306a8126a3dbd841abd62af93954f191dd95e5ea53cd975a77d57bfb8c1 Nov 02 20:35:54.244025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:54.243988 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7"] Nov 02 20:35:55.188313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:55.188272 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" event={"ID":"5c20ca15-e559-4eb5-a92c-c0049a6e0308","Type":"ContainerStarted","Data":"2695c56b339bd07f3129eb09bdda913e811d1103eb54b08ffbdf192768e58a0e"} Nov 02 20:35:55.188718 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:55.188326 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" event={"ID":"5c20ca15-e559-4eb5-a92c-c0049a6e0308","Type":"ContainerStarted","Data":"b411e306a8126a3dbd841abd62af93954f191dd95e5ea53cd975a77d57bfb8c1"} Nov 02 20:35:55.304002 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:55.303953 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podStartSLOduration=2.303938429 podStartE2EDuration="2.303938429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:55.299243464 +0000 UTC m=+1283.588900002" watchObservedRunningTime="2026-11-02 20:35:55.303938429 +0000 UTC m=+1283.593594956" Nov 02 20:35:55.347519 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:55.347499 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr" Nov 02 20:35:55.438482 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:55.438388 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-maas-api-openshift-ca\") pod \"fd8af1ff-e779-4317-b2f3-e9372d6e3a31\" (UID: \"fd8af1ff-e779-4317-b2f3-e9372d6e3a31\") " Nov 02 20:35:55.438793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:55.438766 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-kube-api-access-zsdnm\" (UniqueName: \"kubernetes.io/projected/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-kube-api-access-zsdnm\") pod \"fd8af1ff-e779-4317-b2f3-e9372d6e3a31\" (UID: \"fd8af1ff-e779-4317-b2f3-e9372d6e3a31\") " Nov 02 20:35:55.438953 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:55.438873 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-maas-api-xks-ca\") pod \"fd8af1ff-e779-4317-b2f3-e9372d6e3a31\" (UID: \"fd8af1ff-e779-4317-b2f3-e9372d6e3a31\") " Nov 02 20:35:55.439030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:55.438948 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-maas-api-openshift-ca" pod "fd8af1ff-e779-4317-b2f3-e9372d6e3a31" (UID: "fd8af1ff-e779-4317-b2f3-e9372d6e3a31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:35:55.439493 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:55.439468 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:55.454045 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:55.446269 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-maas-api-xks-ca" pod "fd8af1ff-e779-4317-b2f3-e9372d6e3a31" (UID: "fd8af1ff-e779-4317-b2f3-e9372d6e3a31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:35:55.454045 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:55.446431 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-kube-api-access-zsdnm" pod "fd8af1ff-e779-4317-b2f3-e9372d6e3a31" (UID: "fd8af1ff-e779-4317-b2f3-e9372d6e3a31"). InnerVolumeSpecName "kube-api-access-zsdnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:55.540481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:55.540448 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsdnm\" (UniqueName: \"kubernetes.io/projected/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-kube-api-access-zsdnm\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:55.540481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:55.540476 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8af1ff-e779-4317-b2f3-e9372d6e3a31-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:55.849995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:55.849970 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:35:56.197449 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:56.197380 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr" Nov 02 20:35:56.197449 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:56.197419 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr" event={"ID":"fd8af1ff-e779-4317-b2f3-e9372d6e3a31","Type":"ContainerDied","Data":"5335df24fb988850224ff8e045d29d7ab1a046ea65871b004183b3d4094c2d94"} Nov 02 20:35:56.197864 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:56.197471 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5335df24fb988850224ff8e045d29d7ab1a046ea65871b004183b3d4094c2d94" Nov 02 20:35:57.802464 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:35:57.802426 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:02.057729 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:02.057693 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k"] Nov 02 20:36:02.062299 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:02.062272 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bbb4597-fgq4k"] Nov 02 20:36:02.646021 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:02.645985 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:03.338360 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:36:03.338325 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:36:03.850125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:03.850074 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:36:03.868225 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:03.868196 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc2de4b0-146a-4f77-ac02-43b51df1dea9" path="/var/lib/kubelet/pods/dc2de4b0-146a-4f77-ac02-43b51df1dea9/volumes" Nov 02 20:36:03.868583 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:03.868570 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:36:04.270155 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:04.270066 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:36:05.872315 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:05.872284 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd"] Nov 02 20:36:05.872972 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:05.872958 4971 cpu_manager.go:414] "containerMap: removing container" podUID="fd8af1ff-e779-4317-b2f3-e9372d6e3a31" containerName="revoke-keys" Nov 02 20:36:05.873021 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:05.872973 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8af1ff-e779-4317-b2f3-e9372d6e3a31" containerName="revoke-keys" Nov 02 20:36:05.873113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:05.873076 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8af1ff-e779-4317-b2f3-e9372d6e3a31" containerName="revoke-keys" Nov 02 20:36:05.877991 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:05.877977 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:05.954917 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:05.954876 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:05.954917 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:05.954916 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:05.955203 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:05.955061 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:05.955203 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:05.955123 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f9k9\" (UniqueName: \"kubernetes.io/projected/61a53826-3ab4-4bc6-a88e-dcb48addd13b-kube-api-access-2f9k9\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:05.955343 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:05.955309 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:05.958763 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:05.957866 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:05.958763 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:05.958005 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:05.958763 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:05.958039 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:05.958763 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:05.958121 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.059477 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.059443 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.059477 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.059481 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.059724 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.059539 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.059724 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.059559 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9k9\" (UniqueName: \"kubernetes.io/projected/61a53826-3ab4-4bc6-a88e-dcb48addd13b-kube-api-access-2f9k9\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.059724 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.059599 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.059724 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.059644 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.059938 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.059746 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.059938 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.059781 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.059938 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.059845 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.059938 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.059887 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.060171 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.059968 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.060529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.060346 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.060529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.060464 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.060832 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.060808 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.062047 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.062021 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.062777 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.062755 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.075454 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.075430 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd"] Nov 02 20:36:06.084726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.084703 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx"] Nov 02 20:36:06.089727 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.089710 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" Nov 02 20:36:06.117593 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.117563 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.117706 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.117689 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f9k9\" (UniqueName: \"kubernetes.io/projected/61a53826-3ab4-4bc6-a88e-dcb48addd13b-kube-api-access-2f9k9\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.139537 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.139477 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:36:06.139701 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.139554 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:36:06.142577 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.142554 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx"] Nov 02 20:36:06.160770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.160737 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c558b71-6fgxx\" (UID: \"3bf57a6d-167e-46ce-a278-dfdfa345c3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" Nov 02 20:36:06.160889 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.160855 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4vws\" (UniqueName: \"kubernetes.io/projected/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-kube-api-access-h4vws\") pod \"maas-api-revoke-sub-5c558b71-6fgxx\" (UID: \"3bf57a6d-167e-46ce-a278-dfdfa345c3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" Nov 02 20:36:06.160958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.160902 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c558b71-6fgxx\" (UID: \"3bf57a6d-167e-46ce-a278-dfdfa345c3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" Nov 02 20:36:06.192215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.192191 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:06.262023 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.261998 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c558b71-6fgxx\" (UID: \"3bf57a6d-167e-46ce-a278-dfdfa345c3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" Nov 02 20:36:06.262265 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.262145 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4vws\" (UniqueName: \"kubernetes.io/projected/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-kube-api-access-h4vws\") pod \"maas-api-revoke-sub-5c558b71-6fgxx\" (UID: \"3bf57a6d-167e-46ce-a278-dfdfa345c3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" Nov 02 20:36:06.262265 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.262220 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c558b71-6fgxx\" (UID: \"3bf57a6d-167e-46ce-a278-dfdfa345c3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" Nov 02 20:36:06.263001 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.262970 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c558b71-6fgxx\" (UID: \"3bf57a6d-167e-46ce-a278-dfdfa345c3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" Nov 02 20:36:06.265760 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.265610 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c558b71-6fgxx\" (UID: \"3bf57a6d-167e-46ce-a278-dfdfa345c3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" Nov 02 20:36:06.320602 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.320574 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4vws\" (UniqueName: \"kubernetes.io/projected/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-kube-api-access-h4vws\") pod \"maas-api-revoke-sub-5c558b71-6fgxx\" (UID: \"3bf57a6d-167e-46ce-a278-dfdfa345c3e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" Nov 02 20:36:06.409132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.409055 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" Nov 02 20:36:06.624721 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:36:06.624681 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a53826_3ab4_4bc6_a88e_dcb48addd13b.slice/crio-be566058aac4acc33902e9b1a7e16b91220b5aab06d4ec3863679b42000fed53 WatchSource:0}: Error finding container be566058aac4acc33902e9b1a7e16b91220b5aab06d4ec3863679b42000fed53: Status 404 returned error can't find the container with id be566058aac4acc33902e9b1a7e16b91220b5aab06d4ec3863679b42000fed53 Nov 02 20:36:06.627284 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.627259 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd"] Nov 02 20:36:06.627570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.627536 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:06.627634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.627620 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:06.627672 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.627662 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:06.785953 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:36:06.785888 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf57a6d_167e_46ce_a278_dfdfa345c3e8.slice/crio-430b918f27374231cebb28fd58f14fe6a355843e33d58136fd8987116ca5a5ef WatchSource:0}: Error finding container 430b918f27374231cebb28fd58f14fe6a355843e33d58136fd8987116ca5a5ef: Status 404 returned error can't find the container with id 430b918f27374231cebb28fd58f14fe6a355843e33d58136fd8987116ca5a5ef Nov 02 20:36:06.787936 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:06.787857 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx"] Nov 02 20:36:07.289752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.289666 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" event={"ID":"3bf57a6d-167e-46ce-a278-dfdfa345c3e8","Type":"ContainerStarted","Data":"ff20cd1e2d0341ef9deccdaf4c018ac9c5d4a215f1aa784572d8dc09673b764d"} Nov 02 20:36:07.290201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.289759 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" event={"ID":"3bf57a6d-167e-46ce-a278-dfdfa345c3e8","Type":"ContainerStarted","Data":"430b918f27374231cebb28fd58f14fe6a355843e33d58136fd8987116ca5a5ef"} Nov 02 20:36:07.291559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.291529 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" event={"ID":"61a53826-3ab4-4bc6-a88e-dcb48addd13b","Type":"ContainerStarted","Data":"47c0ab726a8bd644c3ac10e236f633d5ab449d2dd1a3e942b76ec39b057b7922"} Nov 02 20:36:07.291662 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.291575 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" event={"ID":"61a53826-3ab4-4bc6-a88e-dcb48addd13b","Type":"ContainerStarted","Data":"be566058aac4acc33902e9b1a7e16b91220b5aab06d4ec3863679b42000fed53"} Nov 02 20:36:07.407247 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.407194 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" podStartSLOduration=1.407181792 podStartE2EDuration="1.407181792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:07.406456363 +0000 UTC m=+1295.696112891" watchObservedRunningTime="2026-11-02 20:36:07.407181792 +0000 UTC m=+1295.696838319" Nov 02 20:36:07.455487 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.455449 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w"] Nov 02 20:36:07.460282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.460256 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" Nov 02 20:36:07.513215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.513182 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w"] Nov 02 20:36:07.562066 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.561953 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:07.565724 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.565677 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" podStartSLOduration=2.565664489 podStartE2EDuration="2.565664489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:07.563279127 +0000 UTC m=+1295.852935656" watchObservedRunningTime="2026-11-02 20:36:07.565664489 +0000 UTC m=+1295.855321018" Nov 02 20:36:07.581285 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.581255 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbf98b2e-zcl9w\" (UID: \"f9a48e90-9a8b-4a12-95e9-0b5fc5232699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" Nov 02 20:36:07.581459 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.581439 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p65d5\" (UniqueName: \"kubernetes.io/projected/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-kube-api-access-p65d5\") pod \"maas-api-revoke-sub-dbf98b2e-zcl9w\" (UID: \"f9a48e90-9a8b-4a12-95e9-0b5fc5232699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" Nov 02 20:36:07.581539 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.581489 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbf98b2e-zcl9w\" (UID: \"f9a48e90-9a8b-4a12-95e9-0b5fc5232699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" Nov 02 20:36:07.683232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.683196 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p65d5\" (UniqueName: \"kubernetes.io/projected/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-kube-api-access-p65d5\") pod \"maas-api-revoke-sub-dbf98b2e-zcl9w\" (UID: \"f9a48e90-9a8b-4a12-95e9-0b5fc5232699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" Nov 02 20:36:07.683428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.683239 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbf98b2e-zcl9w\" (UID: \"f9a48e90-9a8b-4a12-95e9-0b5fc5232699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" Nov 02 20:36:07.683428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.683389 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbf98b2e-zcl9w\" (UID: \"f9a48e90-9a8b-4a12-95e9-0b5fc5232699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" Nov 02 20:36:07.683952 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.683925 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbf98b2e-zcl9w\" (UID: \"f9a48e90-9a8b-4a12-95e9-0b5fc5232699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" Nov 02 20:36:07.685993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.685972 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbf98b2e-zcl9w\" (UID: \"f9a48e90-9a8b-4a12-95e9-0b5fc5232699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" Nov 02 20:36:07.750666 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.750630 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p65d5\" (UniqueName: \"kubernetes.io/projected/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-kube-api-access-p65d5\") pod \"maas-api-revoke-sub-dbf98b2e-zcl9w\" (UID: \"f9a48e90-9a8b-4a12-95e9-0b5fc5232699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" Nov 02 20:36:07.777979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:07.777955 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" Nov 02 20:36:08.182457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:08.182422 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w"] Nov 02 20:36:08.192501 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:08.192476 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:08.193882 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:08.193855 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 20:36:08.193992 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:08.193912 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" podUID="61a53826-3ab4-4bc6-a88e-dcb48addd13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 20:36:08.302178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:08.302149 4971 generic.go:356] "Generic (PLEG): container finished" podID="3bf57a6d-167e-46ce-a278-dfdfa345c3e8" containerID="ff20cd1e2d0341ef9deccdaf4c018ac9c5d4a215f1aa784572d8dc09673b764d" exitCode=0 Nov 02 20:36:08.302536 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:08.302481 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" event={"ID":"3bf57a6d-167e-46ce-a278-dfdfa345c3e8","Type":"ContainerDied","Data":"ff20cd1e2d0341ef9deccdaf4c018ac9c5d4a215f1aa784572d8dc09673b764d"} Nov 02 20:36:08.303842 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:08.303818 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" event={"ID":"f9a48e90-9a8b-4a12-95e9-0b5fc5232699","Type":"ContainerStarted","Data":"64d4fd796f8e10d50453b16d550d847f080981bd59a00fda0e8f388c93b50128"} Nov 02 20:36:09.192967 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:09.192936 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 20:36:09.193184 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:09.192987 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" podUID="61a53826-3ab4-4bc6-a88e-dcb48addd13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 20:36:09.314355 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:09.314322 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" event={"ID":"f9a48e90-9a8b-4a12-95e9-0b5fc5232699","Type":"ContainerStarted","Data":"87dc388f5c34f8eb70cb644502f4067cf4ab495f382a421354fd463623617228"} Nov 02 20:36:09.458279 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:09.458215 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" podStartSLOduration=2.458200663 podStartE2EDuration="2.458200663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:09.454927941 +0000 UTC m=+1297.744584509" watchObservedRunningTime="2026-11-02 20:36:09.458200663 +0000 UTC m=+1297.747857169" Nov 02 20:36:09.557152 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:09.557127 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" Nov 02 20:36:09.714060 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:09.713963 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-kube-api-access-h4vws\" (UniqueName: \"kubernetes.io/projected/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-kube-api-access-h4vws\") pod \"3bf57a6d-167e-46ce-a278-dfdfa345c3e8\" (UID: \"3bf57a6d-167e-46ce-a278-dfdfa345c3e8\") " Nov 02 20:36:09.714493 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:09.714475 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-maas-api-xks-ca\") pod \"3bf57a6d-167e-46ce-a278-dfdfa345c3e8\" (UID: \"3bf57a6d-167e-46ce-a278-dfdfa345c3e8\") " Nov 02 20:36:09.714566 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:09.714524 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-maas-api-openshift-ca\") pod \"3bf57a6d-167e-46ce-a278-dfdfa345c3e8\" (UID: \"3bf57a6d-167e-46ce-a278-dfdfa345c3e8\") " Nov 02 20:36:09.714907 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:09.714888 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-maas-api-openshift-ca" pod "3bf57a6d-167e-46ce-a278-dfdfa345c3e8" (UID: "3bf57a6d-167e-46ce-a278-dfdfa345c3e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:09.715684 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:09.715657 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:09.717485 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:09.716770 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-kube-api-access-h4vws" pod "3bf57a6d-167e-46ce-a278-dfdfa345c3e8" (UID: "3bf57a6d-167e-46ce-a278-dfdfa345c3e8"). InnerVolumeSpecName "kube-api-access-h4vws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:09.718203 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:09.717755 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-maas-api-xks-ca" pod "3bf57a6d-167e-46ce-a278-dfdfa345c3e8" (UID: "3bf57a6d-167e-46ce-a278-dfdfa345c3e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:09.816973 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:09.816933 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4vws\" (UniqueName: \"kubernetes.io/projected/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-kube-api-access-h4vws\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:09.816973 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:09.816971 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf57a6d-167e-46ce-a278-dfdfa345c3e8-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:10.192886 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:10.192852 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 20:36:10.193101 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:10.192907 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" podUID="61a53826-3ab4-4bc6-a88e-dcb48addd13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 20:36:10.348775 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:10.348745 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" Nov 02 20:36:10.349680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:10.349389 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx" event={"ID":"3bf57a6d-167e-46ce-a278-dfdfa345c3e8","Type":"ContainerDied","Data":"430b918f27374231cebb28fd58f14fe6a355843e33d58136fd8987116ca5a5ef"} Nov 02 20:36:10.349680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:10.349439 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="430b918f27374231cebb28fd58f14fe6a355843e33d58136fd8987116ca5a5ef" Nov 02 20:36:10.351894 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:10.351867 4971 generic.go:356] "Generic (PLEG): container finished" podID="f9a48e90-9a8b-4a12-95e9-0b5fc5232699" containerID="87dc388f5c34f8eb70cb644502f4067cf4ab495f382a421354fd463623617228" exitCode=0 Nov 02 20:36:10.352068 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:10.351988 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" event={"ID":"f9a48e90-9a8b-4a12-95e9-0b5fc5232699","Type":"ContainerDied","Data":"87dc388f5c34f8eb70cb644502f4067cf4ab495f382a421354fd463623617228"} Nov 02 20:36:10.852433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:10.852400 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw"] Nov 02 20:36:10.853291 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:10.853274 4971 cpu_manager.go:414] "containerMap: removing container" podUID="3bf57a6d-167e-46ce-a278-dfdfa345c3e8" containerName="revoke-keys" Nov 02 20:36:10.853414 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:10.853404 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf57a6d-167e-46ce-a278-dfdfa345c3e8" containerName="revoke-keys" Nov 02 20:36:10.853630 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:10.853620 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf57a6d-167e-46ce-a278-dfdfa345c3e8" containerName="revoke-keys" Nov 02 20:36:10.857632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:10.857617 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" Nov 02 20:36:10.903125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:10.902176 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw"] Nov 02 20:36:10.953106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:10.952884 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cba244ab-dwnsw\" (UID: \"6bb55d37-552a-44cc-bc14-0e4c1addfb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" Nov 02 20:36:10.953106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:10.952980 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cba244ab-dwnsw\" (UID: \"6bb55d37-552a-44cc-bc14-0e4c1addfb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" Nov 02 20:36:10.953330 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:10.953148 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j7xw\" (UniqueName: \"kubernetes.io/projected/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-kube-api-access-2j7xw\") pod \"maas-api-revoke-sub-cba244ab-dwnsw\" (UID: \"6bb55d37-552a-44cc-bc14-0e4c1addfb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" Nov 02 20:36:11.054569 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.054541 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cba244ab-dwnsw\" (UID: \"6bb55d37-552a-44cc-bc14-0e4c1addfb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" Nov 02 20:36:11.054759 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.054656 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j7xw\" (UniqueName: \"kubernetes.io/projected/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-kube-api-access-2j7xw\") pod \"maas-api-revoke-sub-cba244ab-dwnsw\" (UID: \"6bb55d37-552a-44cc-bc14-0e4c1addfb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" Nov 02 20:36:11.055114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.054822 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cba244ab-dwnsw\" (UID: \"6bb55d37-552a-44cc-bc14-0e4c1addfb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" Nov 02 20:36:11.055844 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.055801 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cba244ab-dwnsw\" (UID: \"6bb55d37-552a-44cc-bc14-0e4c1addfb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" Nov 02 20:36:11.058645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.058621 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cba244ab-dwnsw\" (UID: \"6bb55d37-552a-44cc-bc14-0e4c1addfb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" Nov 02 20:36:11.114163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.114101 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j7xw\" (UniqueName: \"kubernetes.io/projected/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-kube-api-access-2j7xw\") pod \"maas-api-revoke-sub-cba244ab-dwnsw\" (UID: \"6bb55d37-552a-44cc-bc14-0e4c1addfb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" Nov 02 20:36:11.172543 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.172513 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" Nov 02 20:36:11.192806 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.192778 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 20:36:11.192901 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.192840 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" podUID="61a53826-3ab4-4bc6-a88e-dcb48addd13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 20:36:11.559590 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.559564 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" Nov 02 20:36:11.576382 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:36:11.576351 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb55d37_552a_44cc_bc14_0e4c1addfb3f.slice/crio-00292b33dd27cf5a67b452e3eb007bdd66bbc25dd5abf00164dd9b2f71a69d53 WatchSource:0}: Error finding container 00292b33dd27cf5a67b452e3eb007bdd66bbc25dd5abf00164dd9b2f71a69d53: Status 404 returned error can't find the container with id 00292b33dd27cf5a67b452e3eb007bdd66bbc25dd5abf00164dd9b2f71a69d53 Nov 02 20:36:11.577743 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.577719 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw"] Nov 02 20:36:11.662776 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.662704 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-maas-api-xks-ca\") pod \"f9a48e90-9a8b-4a12-95e9-0b5fc5232699\" (UID: \"f9a48e90-9a8b-4a12-95e9-0b5fc5232699\") " Nov 02 20:36:11.663032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.663019 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-maas-api-openshift-ca\") pod \"f9a48e90-9a8b-4a12-95e9-0b5fc5232699\" (UID: \"f9a48e90-9a8b-4a12-95e9-0b5fc5232699\") " Nov 02 20:36:11.663204 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.663185 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-kube-api-access-p65d5\" (UniqueName: \"kubernetes.io/projected/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-kube-api-access-p65d5\") pod \"f9a48e90-9a8b-4a12-95e9-0b5fc5232699\" (UID: \"f9a48e90-9a8b-4a12-95e9-0b5fc5232699\") " Nov 02 20:36:11.663334 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.663311 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-maas-api-openshift-ca" pod "f9a48e90-9a8b-4a12-95e9-0b5fc5232699" (UID: "f9a48e90-9a8b-4a12-95e9-0b5fc5232699"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:11.663673 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.663654 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:11.664830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.664811 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-maas-api-xks-ca" pod "f9a48e90-9a8b-4a12-95e9-0b5fc5232699" (UID: "f9a48e90-9a8b-4a12-95e9-0b5fc5232699"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:11.665558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.665541 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-kube-api-access-p65d5" pod "f9a48e90-9a8b-4a12-95e9-0b5fc5232699" (UID: "f9a48e90-9a8b-4a12-95e9-0b5fc5232699"). InnerVolumeSpecName "kube-api-access-p65d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:11.765167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.765068 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:11.765167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:11.765122 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p65d5\" (UniqueName: \"kubernetes.io/projected/f9a48e90-9a8b-4a12-95e9-0b5fc5232699-kube-api-access-p65d5\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:12.193149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:12.193118 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 20:36:12.193302 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:12.193171 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" podUID="61a53826-3ab4-4bc6-a88e-dcb48addd13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 20:36:12.376555 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:12.376527 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" Nov 02 20:36:12.376736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:12.376596 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w" event={"ID":"f9a48e90-9a8b-4a12-95e9-0b5fc5232699","Type":"ContainerDied","Data":"64d4fd796f8e10d50453b16d550d847f080981bd59a00fda0e8f388c93b50128"} Nov 02 20:36:12.376736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:12.376652 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64d4fd796f8e10d50453b16d550d847f080981bd59a00fda0e8f388c93b50128" Nov 02 20:36:12.382109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:12.378839 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" event={"ID":"6bb55d37-552a-44cc-bc14-0e4c1addfb3f","Type":"ContainerStarted","Data":"d4e436bdf4c3a97104877ca0e7b511379a9d157f2453350f6b7e0ecc8d802a15"} Nov 02 20:36:12.382109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:12.378908 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" event={"ID":"6bb55d37-552a-44cc-bc14-0e4c1addfb3f","Type":"ContainerStarted","Data":"00292b33dd27cf5a67b452e3eb007bdd66bbc25dd5abf00164dd9b2f71a69d53"} Nov 02 20:36:12.483573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:12.483540 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:12.499385 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:12.499336 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" podStartSLOduration=2.499322627 podStartE2EDuration="2.499322627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:12.495549845 +0000 UTC m=+1300.785206377" watchObservedRunningTime="2026-11-02 20:36:12.499322627 +0000 UTC m=+1300.788979154" Nov 02 20:36:13.193354 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:13.193313 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 20:36:13.193745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:13.193379 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" podUID="61a53826-3ab4-4bc6-a88e-dcb48addd13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 20:36:13.388302 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:13.388263 4971 generic.go:356] "Generic (PLEG): container finished" podID="6bb55d37-552a-44cc-bc14-0e4c1addfb3f" containerID="d4e436bdf4c3a97104877ca0e7b511379a9d157f2453350f6b7e0ecc8d802a15" exitCode=0 Nov 02 20:36:13.388445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:13.388311 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" event={"ID":"6bb55d37-552a-44cc-bc14-0e4c1addfb3f","Type":"ContainerDied","Data":"d4e436bdf4c3a97104877ca0e7b511379a9d157f2453350f6b7e0ecc8d802a15"} Nov 02 20:36:14.197170 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:14.197141 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:14.197615 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:14.197600 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:14.198200 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:14.198171 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:36:14.616833 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:14.616808 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" Nov 02 20:36:14.721746 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:14.721683 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-maas-api-xks-ca\") pod \"6bb55d37-552a-44cc-bc14-0e4c1addfb3f\" (UID: \"6bb55d37-552a-44cc-bc14-0e4c1addfb3f\") " Nov 02 20:36:14.721900 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:14.721751 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-kube-api-access-2j7xw\" (UniqueName: \"kubernetes.io/projected/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-kube-api-access-2j7xw\") pod \"6bb55d37-552a-44cc-bc14-0e4c1addfb3f\" (UID: \"6bb55d37-552a-44cc-bc14-0e4c1addfb3f\") " Nov 02 20:36:14.721900 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:14.721779 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-maas-api-openshift-ca\") pod \"6bb55d37-552a-44cc-bc14-0e4c1addfb3f\" (UID: \"6bb55d37-552a-44cc-bc14-0e4c1addfb3f\") " Nov 02 20:36:14.722206 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:14.722167 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-maas-api-openshift-ca" pod "6bb55d37-552a-44cc-bc14-0e4c1addfb3f" (UID: "6bb55d37-552a-44cc-bc14-0e4c1addfb3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:14.722398 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:14.722379 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:14.724076 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:14.724048 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-maas-api-xks-ca" pod "6bb55d37-552a-44cc-bc14-0e4c1addfb3f" (UID: "6bb55d37-552a-44cc-bc14-0e4c1addfb3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:14.724202 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:14.724167 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-kube-api-access-2j7xw" pod "6bb55d37-552a-44cc-bc14-0e4c1addfb3f" (UID: "6bb55d37-552a-44cc-bc14-0e4c1addfb3f"). InnerVolumeSpecName "kube-api-access-2j7xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:14.824058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:14.824030 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:14.824058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:14.824055 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j7xw\" (UniqueName: \"kubernetes.io/projected/6bb55d37-552a-44cc-bc14-0e4c1addfb3f-kube-api-access-2j7xw\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:15.404709 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.404682 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" Nov 02 20:36:15.405113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.404723 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw" event={"ID":"6bb55d37-552a-44cc-bc14-0e4c1addfb3f","Type":"ContainerDied","Data":"00292b33dd27cf5a67b452e3eb007bdd66bbc25dd5abf00164dd9b2f71a69d53"} Nov 02 20:36:15.405113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.404770 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00292b33dd27cf5a67b452e3eb007bdd66bbc25dd5abf00164dd9b2f71a69d53" Nov 02 20:36:15.901946 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.901918 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v"] Nov 02 20:36:15.902555 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.902538 4971 cpu_manager.go:414] "containerMap: removing container" podUID="f9a48e90-9a8b-4a12-95e9-0b5fc5232699" containerName="revoke-keys" Nov 02 20:36:15.902608 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.902556 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a48e90-9a8b-4a12-95e9-0b5fc5232699" containerName="revoke-keys" Nov 02 20:36:15.902608 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.902570 4971 cpu_manager.go:414] "containerMap: removing container" podUID="6bb55d37-552a-44cc-bc14-0e4c1addfb3f" containerName="revoke-keys" Nov 02 20:36:15.902608 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.902575 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb55d37-552a-44cc-bc14-0e4c1addfb3f" containerName="revoke-keys" Nov 02 20:36:15.902704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.902697 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a48e90-9a8b-4a12-95e9-0b5fc5232699" containerName="revoke-keys" Nov 02 20:36:15.902740 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.902704 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb55d37-552a-44cc-bc14-0e4c1addfb3f" containerName="revoke-keys" Nov 02 20:36:15.906631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.906614 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:15.937421 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.937392 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:15.937560 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.937539 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:15.937639 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.937575 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:15.937639 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.937602 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:15.937719 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.937698 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:15.937830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.937801 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdmf6\" (UniqueName: \"kubernetes.io/projected/fa9309a3-38ea-4870-91dc-2bbab6b13f34-kube-api-access-xdmf6\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:15.937887 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.937870 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:15.937951 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.937933 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:15.938131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.938062 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:15.962823 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:15.962799 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v"] Nov 02 20:36:16.039653 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.039627 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.039793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.039657 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.039793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.039676 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.039793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.039725 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.039922 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.039809 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdmf6\" (UniqueName: \"kubernetes.io/projected/fa9309a3-38ea-4870-91dc-2bbab6b13f34-kube-api-access-xdmf6\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.039922 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.039845 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.039922 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.039871 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.040029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.039980 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.040061 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.040046 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.040111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.040067 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.040430 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.040410 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.040655 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.040632 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.040791 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.040769 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.041364 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.041329 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.044269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.044238 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.044497 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.044476 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.098622 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.098594 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdmf6\" (UniqueName: \"kubernetes.io/projected/fa9309a3-38ea-4870-91dc-2bbab6b13f34-kube-api-access-xdmf6\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.098729 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.098603 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.220572 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.220513 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:16.651327 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:36:16.651297 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa9309a3_38ea_4870_91dc_2bbab6b13f34.slice/crio-9e842f117767348354ec13bd864a66434708ceaa5362b3ac3bad4727e59ee47f WatchSource:0}: Error finding container 9e842f117767348354ec13bd864a66434708ceaa5362b3ac3bad4727e59ee47f: Status 404 returned error can't find the container with id 9e842f117767348354ec13bd864a66434708ceaa5362b3ac3bad4727e59ee47f Nov 02 20:36:16.651966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.651940 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v"] Nov 02 20:36:16.653845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.653813 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:16.653912 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.653877 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:16.653912 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:16.653905 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:17.048831 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:17.048747 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm"] Nov 02 20:36:17.064052 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:17.064002 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9d7e5c-ttpbm"] Nov 02 20:36:17.317974 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:17.317851 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:17.432477 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:17.432444 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" event={"ID":"fa9309a3-38ea-4870-91dc-2bbab6b13f34","Type":"ContainerStarted","Data":"09e05c5dcf8c3580f416e3fbcd96335e567cc6a1f78c4d2aa2e2913c893af1df"} Nov 02 20:36:17.432650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:17.432498 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" event={"ID":"fa9309a3-38ea-4870-91dc-2bbab6b13f34","Type":"ContainerStarted","Data":"9e842f117767348354ec13bd864a66434708ceaa5362b3ac3bad4727e59ee47f"} Nov 02 20:36:17.867770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:17.867741 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db07e7a0-a6ad-4613-9afe-656747fe386d" path="/var/lib/kubelet/pods/db07e7a0-a6ad-4613-9afe-656747fe386d/volumes" Nov 02 20:36:18.221149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:18.221054 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:18.237242 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:18.237205 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 20:36:18.237507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:18.237474 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" podUID="fa9309a3-38ea-4870-91dc-2bbab6b13f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 20:36:19.220816 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:19.220780 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 20:36:19.221247 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:19.220827 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" podUID="fa9309a3-38ea-4870-91dc-2bbab6b13f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 20:36:20.221430 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:20.221395 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 20:36:20.221811 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:20.221450 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" podUID="fa9309a3-38ea-4870-91dc-2bbab6b13f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 20:36:20.978369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:20.978306 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" podStartSLOduration=5.978287198 podStartE2EDuration="5.978287198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:17.555736023 +0000 UTC m=+1305.845392562" watchObservedRunningTime="2026-11-02 20:36:20.978287198 +0000 UTC m=+1309.267943727" Nov 02 20:36:20.989571 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:20.989541 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n"] Nov 02 20:36:21.005098 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.004440 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.084856 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.084814 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n"] Nov 02 20:36:21.111353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.108768 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.111353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.108846 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.111353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.108887 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.111353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.108945 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.111353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.108975 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.111353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.109015 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.111353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.109071 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rptjf\" (UniqueName: \"kubernetes.io/projected/191d909c-208a-43e3-91ad-9d0f979b1cc8-kube-api-access-rptjf\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.111353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.109183 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191d909c-208a-43e3-91ad-9d0f979b1cc8-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.111353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.109252 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.238177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.221065 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191d909c-208a-43e3-91ad-9d0f979b1cc8-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.253299 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.253261 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.253673 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.253653 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.253864 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.253847 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.253993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.253976 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.254151 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.254132 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.279963 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.279840 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.281979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.281953 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.304127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.280501 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.304127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.280932 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.304127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.280722 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.304127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.301916 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.304127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.301999 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rptjf\" (UniqueName: \"kubernetes.io/projected/191d909c-208a-43e3-91ad-9d0f979b1cc8-kube-api-access-rptjf\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.330423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.305016 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 20:36:21.330423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.305065 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" podUID="fa9309a3-38ea-4870-91dc-2bbab6b13f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 20:36:21.336506 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.336478 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191d909c-208a-43e3-91ad-9d0f979b1cc8-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.361121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.360661 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.386451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.385715 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.386451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.386381 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.390476 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.390417 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rptjf\" (UniqueName: \"kubernetes.io/projected/191d909c-208a-43e3-91ad-9d0f979b1cc8-kube-api-access-rptjf\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:21.634110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:21.633361 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:22.536977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:22.520649 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 20:36:22.536977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:22.520720 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" podUID="fa9309a3-38ea-4870-91dc-2bbab6b13f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 20:36:22.782673 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:36:22.782637 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191d909c_208a_43e3_91ad_9d0f979b1cc8.slice/crio-84b802fb7f04b7ef4d7fa98ef99eba2db02c6be641eff57ea8d0267dc9a23afa WatchSource:0}: Error finding container 84b802fb7f04b7ef4d7fa98ef99eba2db02c6be641eff57ea8d0267dc9a23afa: Status 404 returned error can't find the container with id 84b802fb7f04b7ef4d7fa98ef99eba2db02c6be641eff57ea8d0267dc9a23afa Nov 02 20:36:22.886115 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:22.878456 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n"] Nov 02 20:36:22.976929 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:22.890818 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:22.978062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:22.908755 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:22.978337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:22.978304 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:22.978509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:22.978488 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:23.040527 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.040488 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 02 20:36:23.093624 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.093523 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:23.132408 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.117348 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 02 20:36:23.203002 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.202962 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 02 20:36:23.267151 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.206419 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:23.267151 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.206674 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:23.267151 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.225215 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 20:36:23.267151 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.259755 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" podUID="fa9309a3-38ea-4870-91dc-2bbab6b13f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 20:36:23.274111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.270757 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr"] Nov 02 20:36:23.388561 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.388519 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 20:36:23.530110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.523913 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:23.530110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.524064 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 20:36:23.538102 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.535749 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:23.545634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.545584 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:23.736437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.736391 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:23.774200 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.774152 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a370c93c-52pkr"] Nov 02 20:36:23.827110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.806221 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:23.980801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:23.980717 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e2a472-a91f-402d-bf02-3459102e377f" path="/var/lib/kubelet/pods/c9e2a472-a91f-402d-bf02-3459102e377f/volumes" Nov 02 20:36:24.042105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.040845 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:24.042105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.041187 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:36:24.055316 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.055231 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" event={"ID":"191d909c-208a-43e3-91ad-9d0f979b1cc8","Type":"ContainerStarted","Data":"84b802fb7f04b7ef4d7fa98ef99eba2db02c6be641eff57ea8d0267dc9a23afa"} Nov 02 20:36:24.111148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.096405 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 02 20:36:24.111148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.096796 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:36:24.132116 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.132048 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 02 20:36:24.132855 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.132366 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:36:24.245756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.245478 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 20:36:24.245756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.245542 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" podUID="fa9309a3-38ea-4870-91dc-2bbab6b13f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 20:36:24.245756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.245617 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:24.327132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.305692 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:24.327132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.306020 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 02 20:36:24.327132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.306318 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 02 20:36:24.327132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.306607 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:36:24.327132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.306666 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:36:24.327132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.306704 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:36:24.397320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.375706 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:36:24.467128 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.436996 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:24.467128 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.437400 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:36:24.467128 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.438729 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:24.583285 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.583177 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 20:36:24.648108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.644598 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:36:24.648108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.644664 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:36:24.689103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.677492 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded" Nov 02 20:36:24.690162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.689421 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 20:36:24.690310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.690164 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:36:24.690310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.690230 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:36:24.810129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.800669 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:36:24.847130 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.844829 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:24.847130 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.845856 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:36:24.863290 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.856015 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:36:24.885424 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.880608 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:24.903908 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:24.903881 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:36:25.060626 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:25.060567 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:25.078727 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:25.078694 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" event={"ID":"191d909c-208a-43e3-91ad-9d0f979b1cc8","Type":"ContainerStarted","Data":"dc22361ee09fb630cbea7d87d01224409eb15da98838999d775c65ad11d51306"} Nov 02 20:36:25.090681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:25.090650 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:36:25.116870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:25.116787 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:36:25.131012 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:25.130988 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:36:25.157825 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:25.157789 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:36:25.173710 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:25.171026 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:36:25.216993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:25.216960 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:36:25.224945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:25.224218 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 20:36:25.224945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:25.224280 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" podUID="fa9309a3-38ea-4870-91dc-2bbab6b13f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 20:36:25.651117 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:25.650813 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:25.651117 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:25.650963 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 20:36:25.651117 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:25.651013 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" podUID="191d909c-208a-43e3-91ad-9d0f979b1cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 20:36:26.277029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.277004 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:26.277748 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.277719 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:26.280118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.280070 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:36:26.315578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.315534 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" podStartSLOduration=6.315520491 podStartE2EDuration="6.315520491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:26.314946829 +0000 UTC m=+1314.604603357" watchObservedRunningTime="2026-11-02 20:36:26.315520491 +0000 UTC m=+1314.605177018" Nov 02 20:36:26.609863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.609832 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq"] Nov 02 20:36:26.614628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.614607 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" Nov 02 20:36:26.649669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.649642 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 20:36:26.649879 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.649853 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" podUID="191d909c-208a-43e3-91ad-9d0f979b1cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 20:36:26.669611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.669586 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:36:26.674919 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.674881 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:36:26.687566 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.687527 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq"] Nov 02 20:36:26.708051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.708015 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jpfs\" (UniqueName: \"kubernetes.io/projected/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-kube-api-access-9jpfs\") pod \"maas-api-revoke-sub-7272cc2b-g8bwq\" (UID: \"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" Nov 02 20:36:26.708359 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.708335 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7272cc2b-g8bwq\" (UID: \"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" Nov 02 20:36:26.708568 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.708546 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7272cc2b-g8bwq\" (UID: \"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" Nov 02 20:36:26.810573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.810529 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7272cc2b-g8bwq\" (UID: \"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" Nov 02 20:36:26.810747 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.810726 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jpfs\" (UniqueName: \"kubernetes.io/projected/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-kube-api-access-9jpfs\") pod \"maas-api-revoke-sub-7272cc2b-g8bwq\" (UID: \"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" Nov 02 20:36:26.811211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.811185 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7272cc2b-g8bwq\" (UID: \"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" Nov 02 20:36:26.811958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.811931 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7272cc2b-g8bwq\" (UID: \"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" Nov 02 20:36:26.814640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.814617 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7272cc2b-g8bwq\" (UID: \"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" Nov 02 20:36:26.868853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.868787 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jpfs\" (UniqueName: \"kubernetes.io/projected/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-kube-api-access-9jpfs\") pod \"maas-api-revoke-sub-7272cc2b-g8bwq\" (UID: \"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" Nov 02 20:36:26.958431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:26.958398 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" Nov 02 20:36:27.400135 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:36:27.400068 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3781dc44_be73_4b0c_8a66_dcb8c8ffc07c.slice/crio-5eed8ab7b49dabd19c85d9e37d3048ed7b3c88de5e737d069dd16234a835edfd WatchSource:0}: Error finding container 5eed8ab7b49dabd19c85d9e37d3048ed7b3c88de5e737d069dd16234a835edfd: Status 404 returned error can't find the container with id 5eed8ab7b49dabd19c85d9e37d3048ed7b3c88de5e737d069dd16234a835edfd Nov 02 20:36:27.400669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:27.400620 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq"] Nov 02 20:36:27.647187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:27.645409 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 20:36:27.647187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:27.645463 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" podUID="191d909c-208a-43e3-91ad-9d0f979b1cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 20:36:28.019158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.017477 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:28.149987 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.149956 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" event={"ID":"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c","Type":"ContainerStarted","Data":"c5cfc23caa3de755ce27b7e050f661057ce0e7171a01ad2309435d7c66ffeefa"} Nov 02 20:36:28.150317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.150289 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" event={"ID":"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c","Type":"ContainerStarted","Data":"5eed8ab7b49dabd19c85d9e37d3048ed7b3c88de5e737d069dd16234a835edfd"} Nov 02 20:36:28.638103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.635437 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 20:36:28.638103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.635497 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" podUID="191d909c-208a-43e3-91ad-9d0f979b1cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 20:36:28.751171 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.751116 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" podStartSLOduration=2.751102574 podStartE2EDuration="2.751102574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:28.26103723 +0000 UTC m=+1316.550693773" watchObservedRunningTime="2026-11-02 20:36:28.751102574 +0000 UTC m=+1317.040759101" Nov 02 20:36:28.753627 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.753601 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf"] Nov 02 20:36:28.758198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.758170 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" Nov 02 20:36:28.817511 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.817484 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf"] Nov 02 20:36:28.868239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.868214 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c09a3528-b25d-40a2-8df7-14e997368a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8034c8ec-gwhmf\" (UID: \"c09a3528-b25d-40a2-8df7-14e997368a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" Nov 02 20:36:28.868465 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.868447 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c09a3528-b25d-40a2-8df7-14e997368a0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8034c8ec-gwhmf\" (UID: \"c09a3528-b25d-40a2-8df7-14e997368a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" Nov 02 20:36:28.868660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.868650 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmrpj\" (UniqueName: \"kubernetes.io/projected/c09a3528-b25d-40a2-8df7-14e997368a0a-kube-api-access-pmrpj\") pod \"maas-api-revoke-sub-8034c8ec-gwhmf\" (UID: \"c09a3528-b25d-40a2-8df7-14e997368a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" Nov 02 20:36:28.969846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.969756 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c09a3528-b25d-40a2-8df7-14e997368a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8034c8ec-gwhmf\" (UID: \"c09a3528-b25d-40a2-8df7-14e997368a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" Nov 02 20:36:28.969846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.969826 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c09a3528-b25d-40a2-8df7-14e997368a0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8034c8ec-gwhmf\" (UID: \"c09a3528-b25d-40a2-8df7-14e997368a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" Nov 02 20:36:28.970009 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.969967 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrpj\" (UniqueName: \"kubernetes.io/projected/c09a3528-b25d-40a2-8df7-14e997368a0a-kube-api-access-pmrpj\") pod \"maas-api-revoke-sub-8034c8ec-gwhmf\" (UID: \"c09a3528-b25d-40a2-8df7-14e997368a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" Nov 02 20:36:28.970702 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.970677 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c09a3528-b25d-40a2-8df7-14e997368a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8034c8ec-gwhmf\" (UID: \"c09a3528-b25d-40a2-8df7-14e997368a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" Nov 02 20:36:28.972313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:28.972291 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c09a3528-b25d-40a2-8df7-14e997368a0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8034c8ec-gwhmf\" (UID: \"c09a3528-b25d-40a2-8df7-14e997368a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" Nov 02 20:36:29.051288 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:29.051256 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrpj\" (UniqueName: \"kubernetes.io/projected/c09a3528-b25d-40a2-8df7-14e997368a0a-kube-api-access-pmrpj\") pod \"maas-api-revoke-sub-8034c8ec-gwhmf\" (UID: \"c09a3528-b25d-40a2-8df7-14e997368a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" Nov 02 20:36:29.072185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:29.072146 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" Nov 02 20:36:29.075900 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:29.075878 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:29.173516 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:29.173487 4971 generic.go:356] "Generic (PLEG): container finished" podID="3781dc44-be73-4b0c-8a66-dcb8c8ffc07c" containerID="c5cfc23caa3de755ce27b7e050f661057ce0e7171a01ad2309435d7c66ffeefa" exitCode=0 Nov 02 20:36:29.173945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:29.173918 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" event={"ID":"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c","Type":"ContainerDied","Data":"c5cfc23caa3de755ce27b7e050f661057ce0e7171a01ad2309435d7c66ffeefa"} Nov 02 20:36:29.476942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:29.476914 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf"] Nov 02 20:36:29.478374 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:36:29.478344 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc09a3528_b25d_40a2_8df7_14e997368a0a.slice/crio-a84dc8a087f03eec1b29ed5f885b0afbf65943e6d826feefbf6f5b84bfa424d8 WatchSource:0}: Error finding container a84dc8a087f03eec1b29ed5f885b0afbf65943e6d826feefbf6f5b84bfa424d8: Status 404 returned error can't find the container with id a84dc8a087f03eec1b29ed5f885b0afbf65943e6d826feefbf6f5b84bfa424d8 Nov 02 20:36:29.647110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:29.646937 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 20:36:29.647110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:29.646994 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" podUID="191d909c-208a-43e3-91ad-9d0f979b1cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 20:36:30.184622 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.184546 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" event={"ID":"c09a3528-b25d-40a2-8df7-14e997368a0a","Type":"ContainerStarted","Data":"89cd33741715d167584682aa4ed68da5f50bbf5362cd16b2de68456388720737"} Nov 02 20:36:30.184622 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.184598 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" event={"ID":"c09a3528-b25d-40a2-8df7-14e997368a0a","Type":"ContainerStarted","Data":"a84dc8a087f03eec1b29ed5f885b0afbf65943e6d826feefbf6f5b84bfa424d8"} Nov 02 20:36:30.307933 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.307881 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" podStartSLOduration=2.307865532 podStartE2EDuration="2.307865532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:30.306908465 +0000 UTC m=+1318.596564995" watchObservedRunningTime="2026-11-02 20:36:30.307865532 +0000 UTC m=+1318.597522085" Nov 02 20:36:30.390998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.390973 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" Nov 02 20:36:30.499512 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.499430 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-kube-api-access-9jpfs\" (UniqueName: \"kubernetes.io/projected/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-kube-api-access-9jpfs\") pod \"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c\" (UID: \"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c\") " Nov 02 20:36:30.499815 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.499795 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-maas-api-openshift-ca\") pod \"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c\" (UID: \"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c\") " Nov 02 20:36:30.500137 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.500116 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-maas-api-xks-ca\") pod \"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c\" (UID: \"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c\") " Nov 02 20:36:30.500386 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.500163 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-maas-api-openshift-ca" pod "3781dc44-be73-4b0c-8a66-dcb8c8ffc07c" (UID: "3781dc44-be73-4b0c-8a66-dcb8c8ffc07c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:30.501040 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.501025 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:30.501982 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.501962 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-kube-api-access-9jpfs" pod "3781dc44-be73-4b0c-8a66-dcb8c8ffc07c" (UID: "3781dc44-be73-4b0c-8a66-dcb8c8ffc07c"). InnerVolumeSpecName "kube-api-access-9jpfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:30.502265 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.502247 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-maas-api-xks-ca" pod "3781dc44-be73-4b0c-8a66-dcb8c8ffc07c" (UID: "3781dc44-be73-4b0c-8a66-dcb8c8ffc07c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:30.602391 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.602332 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:30.602391 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.602356 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jpfs\" (UniqueName: \"kubernetes.io/projected/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c-kube-api-access-9jpfs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:30.647453 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.640480 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 20:36:30.647453 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.640535 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" podUID="191d909c-208a-43e3-91ad-9d0f979b1cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 20:36:30.843216 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.843137 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f"] Nov 02 20:36:30.843860 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.843845 4971 cpu_manager.go:414] "containerMap: removing container" podUID="3781dc44-be73-4b0c-8a66-dcb8c8ffc07c" containerName="revoke-keys" Nov 02 20:36:30.843910 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.843863 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3781dc44-be73-4b0c-8a66-dcb8c8ffc07c" containerName="revoke-keys" Nov 02 20:36:30.844000 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.843991 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="3781dc44-be73-4b0c-8a66-dcb8c8ffc07c" containerName="revoke-keys" Nov 02 20:36:30.847616 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.847600 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" Nov 02 20:36:30.902888 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.902857 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f"] Nov 02 20:36:30.906638 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.906608 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406ef455-dsl7f\" (UID: \"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" Nov 02 20:36:30.907041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.907016 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406ef455-dsl7f\" (UID: \"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" Nov 02 20:36:30.907335 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:30.907314 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lplt5\" (UniqueName: \"kubernetes.io/projected/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-kube-api-access-lplt5\") pod \"maas-api-revoke-sub-406ef455-dsl7f\" (UID: \"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" Nov 02 20:36:31.008684 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:31.008652 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406ef455-dsl7f\" (UID: \"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" Nov 02 20:36:31.008846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:31.008748 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lplt5\" (UniqueName: \"kubernetes.io/projected/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-kube-api-access-lplt5\") pod \"maas-api-revoke-sub-406ef455-dsl7f\" (UID: \"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" Nov 02 20:36:31.008846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:31.008836 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406ef455-dsl7f\" (UID: \"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" Nov 02 20:36:31.009411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:31.009387 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406ef455-dsl7f\" (UID: \"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" Nov 02 20:36:31.011551 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:31.011528 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406ef455-dsl7f\" (UID: \"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" Nov 02 20:36:31.066586 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:31.066562 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lplt5\" (UniqueName: \"kubernetes.io/projected/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-kube-api-access-lplt5\") pod \"maas-api-revoke-sub-406ef455-dsl7f\" (UID: \"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" Nov 02 20:36:31.163282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:31.163242 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" Nov 02 20:36:31.195158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:31.195130 4971 generic.go:356] "Generic (PLEG): container finished" podID="c09a3528-b25d-40a2-8df7-14e997368a0a" containerID="89cd33741715d167584682aa4ed68da5f50bbf5362cd16b2de68456388720737" exitCode=0 Nov 02 20:36:31.195531 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:31.195217 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" event={"ID":"c09a3528-b25d-40a2-8df7-14e997368a0a","Type":"ContainerDied","Data":"89cd33741715d167584682aa4ed68da5f50bbf5362cd16b2de68456388720737"} Nov 02 20:36:31.196884 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:31.196854 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" Nov 02 20:36:31.196998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:31.196898 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq" event={"ID":"3781dc44-be73-4b0c-8a66-dcb8c8ffc07c","Type":"ContainerDied","Data":"5eed8ab7b49dabd19c85d9e37d3048ed7b3c88de5e737d069dd16234a835edfd"} Nov 02 20:36:31.196998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:31.196949 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eed8ab7b49dabd19c85d9e37d3048ed7b3c88de5e737d069dd16234a835edfd" Nov 02 20:36:31.552291 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:36:31.552260 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eaf0d3d_a203_4b8d_ab64_5d6a5f114ce9.slice/crio-1960e421132550f0b26f5b28f4ae9104c816254f0b521a1c7749aabd16b3350b WatchSource:0}: Error finding container 1960e421132550f0b26f5b28f4ae9104c816254f0b521a1c7749aabd16b3350b: Status 404 returned error can't find the container with id 1960e421132550f0b26f5b28f4ae9104c816254f0b521a1c7749aabd16b3350b Nov 02 20:36:31.556677 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:31.556606 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f"] Nov 02 20:36:31.634041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:31.634018 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 02 20:36:31.634179 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:31.634063 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" podUID="191d909c-208a-43e3-91ad-9d0f979b1cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 02 20:36:32.135246 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.135213 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw"] Nov 02 20:36:32.142434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.142407 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw" Nov 02 20:36:32.190162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.190139 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw"] Nov 02 20:36:32.219117 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.213402 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" event={"ID":"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9","Type":"ContainerStarted","Data":"b2009b532549c3410193e9776302f8c593fa0c889960e778a293a096d3fb21b8"} Nov 02 20:36:32.219117 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.213467 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" event={"ID":"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9","Type":"ContainerStarted","Data":"1960e421132550f0b26f5b28f4ae9104c816254f0b521a1c7749aabd16b3350b"} Nov 02 20:36:32.237247 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.236654 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a4cb21-606f-4b21-819d-9e312d0fce5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a1ba2f-nk8cw\" (UID: \"60a4cb21-606f-4b21-819d-9e312d0fce5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw" Nov 02 20:36:32.237247 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.236788 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a4cb21-606f-4b21-819d-9e312d0fce5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a1ba2f-nk8cw\" (UID: \"60a4cb21-606f-4b21-819d-9e312d0fce5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw" Nov 02 20:36:32.237247 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.236933 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cphdz\" (UniqueName: \"kubernetes.io/projected/60a4cb21-606f-4b21-819d-9e312d0fce5f-kube-api-access-cphdz\") pod \"maas-api-revoke-sub-c2a1ba2f-nk8cw\" (UID: \"60a4cb21-606f-4b21-819d-9e312d0fce5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw" Nov 02 20:36:32.338763 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.338736 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a4cb21-606f-4b21-819d-9e312d0fce5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a1ba2f-nk8cw\" (UID: \"60a4cb21-606f-4b21-819d-9e312d0fce5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw" Nov 02 20:36:32.338893 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.338864 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cphdz\" (UniqueName: \"kubernetes.io/projected/60a4cb21-606f-4b21-819d-9e312d0fce5f-kube-api-access-cphdz\") pod \"maas-api-revoke-sub-c2a1ba2f-nk8cw\" (UID: \"60a4cb21-606f-4b21-819d-9e312d0fce5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw" Nov 02 20:36:32.338952 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.338928 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a4cb21-606f-4b21-819d-9e312d0fce5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a1ba2f-nk8cw\" (UID: \"60a4cb21-606f-4b21-819d-9e312d0fce5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw" Nov 02 20:36:32.339674 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.339644 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a4cb21-606f-4b21-819d-9e312d0fce5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a1ba2f-nk8cw\" (UID: \"60a4cb21-606f-4b21-819d-9e312d0fce5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw" Nov 02 20:36:32.341674 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.341621 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a4cb21-606f-4b21-819d-9e312d0fce5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a1ba2f-nk8cw\" (UID: \"60a4cb21-606f-4b21-819d-9e312d0fce5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw" Nov 02 20:36:32.366595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.366553 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" podStartSLOduration=2.366538362 podStartE2EDuration="2.366538362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:32.363483804 +0000 UTC m=+1320.653140333" watchObservedRunningTime="2026-11-02 20:36:32.366538362 +0000 UTC m=+1320.656194890" Nov 02 20:36:32.396921 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.396896 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cphdz\" (UniqueName: \"kubernetes.io/projected/60a4cb21-606f-4b21-819d-9e312d0fce5f-kube-api-access-cphdz\") pod \"maas-api-revoke-sub-c2a1ba2f-nk8cw\" (UID: \"60a4cb21-606f-4b21-819d-9e312d0fce5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw" Nov 02 20:36:32.419625 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.419607 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" Nov 02 20:36:32.456732 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.456712 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw" Nov 02 20:36:32.541735 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.541711 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c09a3528-b25d-40a2-8df7-14e997368a0a-kube-api-access-pmrpj\" (UniqueName: \"kubernetes.io/projected/c09a3528-b25d-40a2-8df7-14e997368a0a-kube-api-access-pmrpj\") pod \"c09a3528-b25d-40a2-8df7-14e997368a0a\" (UID: \"c09a3528-b25d-40a2-8df7-14e997368a0a\") " Nov 02 20:36:32.541896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.541747 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c09a3528-b25d-40a2-8df7-14e997368a0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c09a3528-b25d-40a2-8df7-14e997368a0a-maas-api-xks-ca\") pod \"c09a3528-b25d-40a2-8df7-14e997368a0a\" (UID: \"c09a3528-b25d-40a2-8df7-14e997368a0a\") " Nov 02 20:36:32.542014 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.541977 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c09a3528-b25d-40a2-8df7-14e997368a0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c09a3528-b25d-40a2-8df7-14e997368a0a-maas-api-openshift-ca\") pod \"c09a3528-b25d-40a2-8df7-14e997368a0a\" (UID: \"c09a3528-b25d-40a2-8df7-14e997368a0a\") " Nov 02 20:36:32.542968 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.542533 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c09a3528-b25d-40a2-8df7-14e997368a0a-maas-api-openshift-ca" pod "c09a3528-b25d-40a2-8df7-14e997368a0a" (UID: "c09a3528-b25d-40a2-8df7-14e997368a0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:32.542968 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.542935 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c09a3528-b25d-40a2-8df7-14e997368a0a-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:32.545326 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.545299 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c09a3528-b25d-40a2-8df7-14e997368a0a-maas-api-xks-ca" pod "c09a3528-b25d-40a2-8df7-14e997368a0a" (UID: "c09a3528-b25d-40a2-8df7-14e997368a0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:32.545481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.545303 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09a3528-b25d-40a2-8df7-14e997368a0a-kube-api-access-pmrpj" pod "c09a3528-b25d-40a2-8df7-14e997368a0a" (UID: "c09a3528-b25d-40a2-8df7-14e997368a0a"). InnerVolumeSpecName "kube-api-access-pmrpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:32.639792 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.639730 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:32.640224 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.640210 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:32.640817 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.640801 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:36:32.644729 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.644709 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmrpj\" (UniqueName: \"kubernetes.io/projected/c09a3528-b25d-40a2-8df7-14e997368a0a-kube-api-access-pmrpj\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:32.644798 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.644734 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c09a3528-b25d-40a2-8df7-14e997368a0a-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:32.871112 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.867013 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw"] Nov 02 20:36:32.933417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:32.933386 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:33.055921 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:33.055879 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf"] Nov 02 20:36:33.061243 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:33.061217 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c209ba40-ln4bf"] Nov 02 20:36:33.224668 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:33.224589 4971 generic.go:356] "Generic (PLEG): container finished" podID="2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9" containerID="b2009b532549c3410193e9776302f8c593fa0c889960e778a293a096d3fb21b8" exitCode=0 Nov 02 20:36:33.225255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:33.224693 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" event={"ID":"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9","Type":"ContainerDied","Data":"b2009b532549c3410193e9776302f8c593fa0c889960e778a293a096d3fb21b8"} Nov 02 20:36:33.226413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:33.226384 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw" event={"ID":"60a4cb21-606f-4b21-819d-9e312d0fce5f","Type":"ContainerStarted","Data":"fe99061771be11abe12596b8950eb8aec341163e6e432ad3c7a37c5d8136683c"} Nov 02 20:36:33.228136 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:33.228110 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" event={"ID":"c09a3528-b25d-40a2-8df7-14e997368a0a","Type":"ContainerDied","Data":"a84dc8a087f03eec1b29ed5f885b0afbf65943e6d826feefbf6f5b84bfa424d8"} Nov 02 20:36:33.228238 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:33.228157 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf" Nov 02 20:36:33.228238 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:33.228169 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a84dc8a087f03eec1b29ed5f885b0afbf65943e6d826feefbf6f5b84bfa424d8" Nov 02 20:36:33.341126 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:36:33.341069 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:36:33.868812 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:33.868785 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e930cdea-4538-4672-a8a2-bf16d01ffb64" path="/var/lib/kubelet/pods/e930cdea-4538-4672-a8a2-bf16d01ffb64/volumes" Nov 02 20:36:34.236935 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:34.236903 4971 generic.go:356] "Generic (PLEG): container finished" podID="60a4cb21-606f-4b21-819d-9e312d0fce5f" containerID="0d1563d72abf61e6a47babd707db6d8c2048d7e440f17fced6a8be58d7eeafa6" exitCode=0 Nov 02 20:36:34.237409 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:34.236982 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw" event={"ID":"60a4cb21-606f-4b21-819d-9e312d0fce5f","Type":"ContainerDied","Data":"0d1563d72abf61e6a47babd707db6d8c2048d7e440f17fced6a8be58d7eeafa6"} Nov 02 20:36:34.426315 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:34.426292 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" Nov 02 20:36:34.575710 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:34.575629 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-kube-api-access-lplt5\" (UniqueName: \"kubernetes.io/projected/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-kube-api-access-lplt5\") pod \"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9\" (UID: \"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9\") " Nov 02 20:36:34.575952 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:34.575916 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-maas-api-openshift-ca\") pod \"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9\" (UID: \"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9\") " Nov 02 20:36:34.576035 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:34.576003 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-maas-api-xks-ca\") pod \"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9\" (UID: \"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9\") " Nov 02 20:36:34.576300 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:34.576279 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-maas-api-openshift-ca" pod "2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9" (UID: "2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:34.576653 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:34.576634 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:34.578498 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:34.578471 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-kube-api-access-lplt5" pod "2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9" (UID: "2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9"). InnerVolumeSpecName "kube-api-access-lplt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:34.578699 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:34.578678 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-maas-api-xks-ca" pod "2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9" (UID: "2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:34.678149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:34.678119 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lplt5\" (UniqueName: \"kubernetes.io/projected/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-kube-api-access-lplt5\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:34.678260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:34.678149 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:35.152939 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.152902 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p"] Nov 02 20:36:35.153616 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.153599 4971 cpu_manager.go:414] "containerMap: removing container" podUID="2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9" containerName="revoke-keys" Nov 02 20:36:35.153688 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.153618 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9" containerName="revoke-keys" Nov 02 20:36:35.153688 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.153636 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c09a3528-b25d-40a2-8df7-14e997368a0a" containerName="revoke-keys" Nov 02 20:36:35.153688 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.153640 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c09a3528-b25d-40a2-8df7-14e997368a0a" containerName="revoke-keys" Nov 02 20:36:35.153786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.153749 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9" containerName="revoke-keys" Nov 02 20:36:35.153786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.153759 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c09a3528-b25d-40a2-8df7-14e997368a0a" containerName="revoke-keys" Nov 02 20:36:35.157588 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.157568 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" Nov 02 20:36:35.210932 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.210907 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p"] Nov 02 20:36:35.247610 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.247592 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" Nov 02 20:36:35.248003 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.247629 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f" event={"ID":"2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9","Type":"ContainerDied","Data":"1960e421132550f0b26f5b28f4ae9104c816254f0b521a1c7749aabd16b3350b"} Nov 02 20:36:35.248003 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.247683 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1960e421132550f0b26f5b28f4ae9104c816254f0b521a1c7749aabd16b3350b" Nov 02 20:36:35.286827 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.286791 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a94a51-90f5-4f9b-9118-3c1a85eb8690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef946bb-ccq6p\" (UID: \"67a94a51-90f5-4f9b-9118-3c1a85eb8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" Nov 02 20:36:35.287069 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.287045 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v86h\" (UniqueName: \"kubernetes.io/projected/67a94a51-90f5-4f9b-9118-3c1a85eb8690-kube-api-access-5v86h\") pod \"maas-api-revoke-sub-4ef946bb-ccq6p\" (UID: \"67a94a51-90f5-4f9b-9118-3c1a85eb8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" Nov 02 20:36:35.287142 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.287121 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a94a51-90f5-4f9b-9118-3c1a85eb8690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef946bb-ccq6p\" (UID: \"67a94a51-90f5-4f9b-9118-3c1a85eb8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" Nov 02 20:36:35.388492 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.388460 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a94a51-90f5-4f9b-9118-3c1a85eb8690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef946bb-ccq6p\" (UID: \"67a94a51-90f5-4f9b-9118-3c1a85eb8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" Nov 02 20:36:35.388671 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.388646 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v86h\" (UniqueName: \"kubernetes.io/projected/67a94a51-90f5-4f9b-9118-3c1a85eb8690-kube-api-access-5v86h\") pod \"maas-api-revoke-sub-4ef946bb-ccq6p\" (UID: \"67a94a51-90f5-4f9b-9118-3c1a85eb8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" Nov 02 20:36:35.388803 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.388698 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a94a51-90f5-4f9b-9118-3c1a85eb8690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef946bb-ccq6p\" (UID: \"67a94a51-90f5-4f9b-9118-3c1a85eb8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" Nov 02 20:36:35.389848 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.389825 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a94a51-90f5-4f9b-9118-3c1a85eb8690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef946bb-ccq6p\" (UID: \"67a94a51-90f5-4f9b-9118-3c1a85eb8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" Nov 02 20:36:35.391008 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.390986 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a94a51-90f5-4f9b-9118-3c1a85eb8690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef946bb-ccq6p\" (UID: \"67a94a51-90f5-4f9b-9118-3c1a85eb8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" Nov 02 20:36:35.416373 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.416323 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw" Nov 02 20:36:35.447323 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.447297 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v86h\" (UniqueName: \"kubernetes.io/projected/67a94a51-90f5-4f9b-9118-3c1a85eb8690-kube-api-access-5v86h\") pod \"maas-api-revoke-sub-4ef946bb-ccq6p\" (UID: \"67a94a51-90f5-4f9b-9118-3c1a85eb8690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" Nov 02 20:36:35.475266 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.475246 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" Nov 02 20:36:35.590961 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.590923 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60a4cb21-606f-4b21-819d-9e312d0fce5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a4cb21-606f-4b21-819d-9e312d0fce5f-maas-api-openshift-ca\") pod \"60a4cb21-606f-4b21-819d-9e312d0fce5f\" (UID: \"60a4cb21-606f-4b21-819d-9e312d0fce5f\") " Nov 02 20:36:35.591150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.591122 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60a4cb21-606f-4b21-819d-9e312d0fce5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a4cb21-606f-4b21-819d-9e312d0fce5f-maas-api-xks-ca\") pod \"60a4cb21-606f-4b21-819d-9e312d0fce5f\" (UID: \"60a4cb21-606f-4b21-819d-9e312d0fce5f\") " Nov 02 20:36:35.591250 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.591234 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a4cb21-606f-4b21-819d-9e312d0fce5f-kube-api-access-cphdz\" (UniqueName: \"kubernetes.io/projected/60a4cb21-606f-4b21-819d-9e312d0fce5f-kube-api-access-cphdz\") pod \"60a4cb21-606f-4b21-819d-9e312d0fce5f\" (UID: \"60a4cb21-606f-4b21-819d-9e312d0fce5f\") " Nov 02 20:36:35.591348 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.591319 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a4cb21-606f-4b21-819d-9e312d0fce5f-maas-api-openshift-ca" pod "60a4cb21-606f-4b21-819d-9e312d0fce5f" (UID: "60a4cb21-606f-4b21-819d-9e312d0fce5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:35.591826 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.591757 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a4cb21-606f-4b21-819d-9e312d0fce5f-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:35.593658 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.593631 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a4cb21-606f-4b21-819d-9e312d0fce5f-maas-api-xks-ca" pod "60a4cb21-606f-4b21-819d-9e312d0fce5f" (UID: "60a4cb21-606f-4b21-819d-9e312d0fce5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:35.593768 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.593733 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a4cb21-606f-4b21-819d-9e312d0fce5f-kube-api-access-cphdz" pod "60a4cb21-606f-4b21-819d-9e312d0fce5f" (UID: "60a4cb21-606f-4b21-819d-9e312d0fce5f"). InnerVolumeSpecName "kube-api-access-cphdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:35.692840 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.692778 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a4cb21-606f-4b21-819d-9e312d0fce5f-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:35.692840 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.692805 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cphdz\" (UniqueName: \"kubernetes.io/projected/60a4cb21-606f-4b21-819d-9e312d0fce5f-kube-api-access-cphdz\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:35.870248 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:36:35.870211 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a94a51_90f5_4f9b_9118_3c1a85eb8690.slice/crio-6bd1d5885a9e99b5121c401cedbced1b5982c7c4a4d2e66a5444266d9e8df8bc WatchSource:0}: Error finding container 6bd1d5885a9e99b5121c401cedbced1b5982c7c4a4d2e66a5444266d9e8df8bc: Status 404 returned error can't find the container with id 6bd1d5885a9e99b5121c401cedbced1b5982c7c4a4d2e66a5444266d9e8df8bc Nov 02 20:36:35.875980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:35.875958 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p"] Nov 02 20:36:36.256772 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.256742 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw" Nov 02 20:36:36.257257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.256742 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw" event={"ID":"60a4cb21-606f-4b21-819d-9e312d0fce5f","Type":"ContainerDied","Data":"fe99061771be11abe12596b8950eb8aec341163e6e432ad3c7a37c5d8136683c"} Nov 02 20:36:36.257257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.256898 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe99061771be11abe12596b8950eb8aec341163e6e432ad3c7a37c5d8136683c" Nov 02 20:36:36.259290 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.259125 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" event={"ID":"67a94a51-90f5-4f9b-9118-3c1a85eb8690","Type":"ContainerStarted","Data":"2c8b904c1756060e8fcd3afcca3dbb505756cecf2e23e0ed7a8cf41bf3b8d09b"} Nov 02 20:36:36.259290 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.259180 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" event={"ID":"67a94a51-90f5-4f9b-9118-3c1a85eb8690","Type":"ContainerStarted","Data":"6bd1d5885a9e99b5121c401cedbced1b5982c7c4a4d2e66a5444266d9e8df8bc"} Nov 02 20:36:36.369559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.369504 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" podStartSLOduration=1.369489326 podStartE2EDuration="1.369489326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:36.367136561 +0000 UTC m=+1324.656793090" watchObservedRunningTime="2026-11-02 20:36:36.369489326 +0000 UTC m=+1324.659145853" Nov 02 20:36:36.419663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.419633 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624"] Nov 02 20:36:36.420374 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.420359 4971 cpu_manager.go:414] "containerMap: removing container" podUID="60a4cb21-606f-4b21-819d-9e312d0fce5f" containerName="revoke-keys" Nov 02 20:36:36.420431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.420377 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a4cb21-606f-4b21-819d-9e312d0fce5f" containerName="revoke-keys" Nov 02 20:36:36.420522 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.420512 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a4cb21-606f-4b21-819d-9e312d0fce5f" containerName="revoke-keys" Nov 02 20:36:36.424428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.424408 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.479660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.479635 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624"] Nov 02 20:36:36.503064 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.503035 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.503186 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.503114 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.503293 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.503269 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.503356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.503332 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714bd47a-f22d-42ac-9692-b6080d740fc0-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.503551 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.503525 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.503604 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.503577 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.503695 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.503677 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.503759 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.503740 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75bq9\" (UniqueName: \"kubernetes.io/projected/714bd47a-f22d-42ac-9692-b6080d740fc0-kube-api-access-75bq9\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.503847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.503827 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.605205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.605172 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.605349 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.605213 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.605349 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.605267 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.605349 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.605319 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75bq9\" (UniqueName: \"kubernetes.io/projected/714bd47a-f22d-42ac-9692-b6080d740fc0-kube-api-access-75bq9\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.605524 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.605364 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.605524 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.605449 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.605524 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.605495 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.605676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.605578 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.605676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.605628 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714bd47a-f22d-42ac-9692-b6080d740fc0-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.605780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.605677 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.605780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.605702 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.605871 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.605801 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.605922 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.605876 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.606304 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.606285 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714bd47a-f22d-42ac-9692-b6080d740fc0-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.607878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.607860 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.607956 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.607941 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.663515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.663437 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.663515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.663460 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75bq9\" (UniqueName: \"kubernetes.io/projected/714bd47a-f22d-42ac-9692-b6080d740fc0-kube-api-access-75bq9\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.738853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.738831 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:36.986314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.986239 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75"] Nov 02 20:36:36.991474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:36.991450 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.058585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.058557 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75"] Nov 02 20:36:37.113605 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.113581 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-istio-data\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.113778 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.113612 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-workload-certs\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.113778 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.113648 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0be871c-832b-4e45-965c-d092b43030eb-istio-podinfo\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.113957 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.113934 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-istio-envoy\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.114022 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.113975 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-workload-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.114123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.114064 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0be871c-832b-4e45-965c-d092b43030eb-istio-token\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.114182 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.114161 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vbn4\" (UniqueName: \"kubernetes.io/projected/f0be871c-832b-4e45-965c-d092b43030eb-kube-api-access-8vbn4\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.114248 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.114211 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-credential-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.114287 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.114248 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0be871c-832b-4e45-965c-d092b43030eb-istiod-ca-cert\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.215050 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.215019 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-istio-envoy\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.215227 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.215057 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-workload-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.215227 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.215130 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0be871c-832b-4e45-965c-d092b43030eb-istio-token\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.215227 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.215195 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vbn4\" (UniqueName: \"kubernetes.io/projected/f0be871c-832b-4e45-965c-d092b43030eb-kube-api-access-8vbn4\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.215397 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.215232 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-credential-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.215397 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.215257 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0be871c-832b-4e45-965c-d092b43030eb-istiod-ca-cert\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.215397 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.215329 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-istio-data\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.215397 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.215359 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-workload-certs\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.215601 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.215416 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0be871c-832b-4e45-965c-d092b43030eb-istio-podinfo\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.215601 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.215527 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-workload-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.215711 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.215592 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-credential-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.215769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.215711 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-istio-data\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.215847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.215826 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-workload-certs\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.216035 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.216014 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0be871c-832b-4e45-965c-d092b43030eb-istiod-ca-cert\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.217634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.217617 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-istio-envoy\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.218305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.218281 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0be871c-832b-4e45-965c-d092b43030eb-istio-podinfo\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.225134 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.225027 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624"] Nov 02 20:36:37.227943 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:36:37.227921 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714bd47a_f22d_42ac_9692_b6080d740fc0.slice/crio-ed93e058e9ffffba03f5087c3cce0e7b334f055d1777756bbf7d939b257c22f4 WatchSource:0}: Error finding container ed93e058e9ffffba03f5087c3cce0e7b334f055d1777756bbf7d939b257c22f4: Status 404 returned error can't find the container with id ed93e058e9ffffba03f5087c3cce0e7b334f055d1777756bbf7d939b257c22f4 Nov 02 20:36:37.230482 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.230453 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:37.230564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.230519 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:37.230604 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.230566 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:37.269484 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.269461 4971 generic.go:356] "Generic (PLEG): container finished" podID="67a94a51-90f5-4f9b-9118-3c1a85eb8690" containerID="2c8b904c1756060e8fcd3afcca3dbb505756cecf2e23e0ed7a8cf41bf3b8d09b" exitCode=0 Nov 02 20:36:37.269828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.269613 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" event={"ID":"67a94a51-90f5-4f9b-9118-3c1a85eb8690","Type":"ContainerDied","Data":"2c8b904c1756060e8fcd3afcca3dbb505756cecf2e23e0ed7a8cf41bf3b8d09b"} Nov 02 20:36:37.271419 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.271396 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" event={"ID":"714bd47a-f22d-42ac-9692-b6080d740fc0","Type":"ContainerStarted","Data":"ed93e058e9ffffba03f5087c3cce0e7b334f055d1777756bbf7d939b257c22f4"} Nov 02 20:36:37.276521 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.276500 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vbn4\" (UniqueName: \"kubernetes.io/projected/f0be871c-832b-4e45-965c-d092b43030eb-kube-api-access-8vbn4\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.277510 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.277493 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0be871c-832b-4e45-965c-d092b43030eb-istio-token\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.308645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.308616 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:37.348363 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.348323 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh"] Nov 02 20:36:37.361101 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.360438 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.426974 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.417825 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh"] Nov 02 20:36:37.520306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.520204 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.520457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.520385 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.520516 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.520477 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.520589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.520562 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.520650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.520612 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea994d9c-d955-4efa-9eea-9225a6fee64a-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.520706 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.520644 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.520781 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.520765 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.520914 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.520886 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.521003 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.520927 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fthpx\" (UniqueName: \"kubernetes.io/projected/ea994d9c-d955-4efa-9eea-9225a6fee64a-kube-api-access-fthpx\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.622947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.622914 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.623193 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.623059 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.623193 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.623107 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.623193 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.623140 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea994d9c-d955-4efa-9eea-9225a6fee64a-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.623193 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.623162 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.623429 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.623237 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.623429 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.623366 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.623429 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.623372 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.623429 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.623400 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fthpx\" (UniqueName: \"kubernetes.io/projected/ea994d9c-d955-4efa-9eea-9225a6fee64a-kube-api-access-fthpx\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.623611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.623445 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.623611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.623451 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.623611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.623478 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.623759 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.623733 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.623964 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.623938 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea994d9c-d955-4efa-9eea-9225a6fee64a-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.625636 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.625608 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.626064 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.626046 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.680888 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.680857 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.681016 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.680896 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fthpx\" (UniqueName: \"kubernetes.io/projected/ea994d9c-d955-4efa-9eea-9225a6fee64a-kube-api-access-fthpx\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.682847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.682829 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:37.722824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.722788 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75"] Nov 02 20:36:37.747904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.747864 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:37.748015 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.747944 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:37.748015 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.747977 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:37.780431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:37.780275 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:38.120112 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.120069 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh"] Nov 02 20:36:38.129013 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.128973 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:38.129248 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.129223 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:38.129405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.129387 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:36:38.287044 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.287006 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" event={"ID":"ea994d9c-d955-4efa-9eea-9225a6fee64a","Type":"ContainerStarted","Data":"46935c5f2a2160f0bb05face02646a3f8d5e21a3337f943d2a32e46a7c917e0d"} Nov 02 20:36:38.287467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.287074 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" event={"ID":"ea994d9c-d955-4efa-9eea-9225a6fee64a","Type":"ContainerStarted","Data":"eb54c54072cdc3df4ff15d58bbec2a4e3329aa250524b21ac56d031504de742b"} Nov 02 20:36:38.289712 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.289683 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" event={"ID":"f0be871c-832b-4e45-965c-d092b43030eb","Type":"ContainerStarted","Data":"f56e4dddc8c2f2f840507acd69acac1751371c4a8e581c1eed6d2dade80389fd"} Nov 02 20:36:38.289765 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.289742 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" event={"ID":"f0be871c-832b-4e45-965c-d092b43030eb","Type":"ContainerStarted","Data":"eb3362f0afc128396a452395d4a2456390bfd1ff45f4304dcfc17f4a3a170392"} Nov 02 20:36:38.291853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.291805 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" event={"ID":"714bd47a-f22d-42ac-9692-b6080d740fc0","Type":"ContainerStarted","Data":"ebdc4dadcf213cf5563e8e13a1f429ab72fef9af5fad134765aeee8618c2816d"} Nov 02 20:36:38.309304 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.309280 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:38.311051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.311020 4971 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:36:38.311212 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.311072 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" podUID="f0be871c-832b-4e45-965c-d092b43030eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 20:36:38.420471 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.420409 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" podStartSLOduration=1.420386742 podStartE2EDuration="1.420386742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:38.401360543 +0000 UTC m=+1326.691017071" watchObservedRunningTime="2026-11-02 20:36:38.420386742 +0000 UTC m=+1326.710043271" Nov 02 20:36:38.593669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.590402 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" podStartSLOduration=2.590384712 podStartE2EDuration="2.590384712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:38.573815093 +0000 UTC m=+1326.863471624" watchObservedRunningTime="2026-11-02 20:36:38.590384712 +0000 UTC m=+1326.880041240" Nov 02 20:36:38.593669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.593581 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" Nov 02 20:36:38.676563 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.676457 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a94a51-90f5-4f9b-9118-3c1a85eb8690-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a94a51-90f5-4f9b-9118-3c1a85eb8690-maas-api-openshift-ca\") pod \"67a94a51-90f5-4f9b-9118-3c1a85eb8690\" (UID: \"67a94a51-90f5-4f9b-9118-3c1a85eb8690\") " Nov 02 20:36:38.678866 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.677413 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a94a51-90f5-4f9b-9118-3c1a85eb8690-kube-api-access-5v86h\" (UniqueName: \"kubernetes.io/projected/67a94a51-90f5-4f9b-9118-3c1a85eb8690-kube-api-access-5v86h\") pod \"67a94a51-90f5-4f9b-9118-3c1a85eb8690\" (UID: \"67a94a51-90f5-4f9b-9118-3c1a85eb8690\") " Nov 02 20:36:38.678866 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.677579 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67a94a51-90f5-4f9b-9118-3c1a85eb8690-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a94a51-90f5-4f9b-9118-3c1a85eb8690-maas-api-xks-ca\") pod \"67a94a51-90f5-4f9b-9118-3c1a85eb8690\" (UID: \"67a94a51-90f5-4f9b-9118-3c1a85eb8690\") " Nov 02 20:36:38.678866 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.678303 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a94a51-90f5-4f9b-9118-3c1a85eb8690-maas-api-openshift-ca" pod "67a94a51-90f5-4f9b-9118-3c1a85eb8690" (UID: "67a94a51-90f5-4f9b-9118-3c1a85eb8690"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:38.681145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.681118 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a94a51-90f5-4f9b-9118-3c1a85eb8690-maas-api-xks-ca" pod "67a94a51-90f5-4f9b-9118-3c1a85eb8690" (UID: "67a94a51-90f5-4f9b-9118-3c1a85eb8690"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:38.681259 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.681197 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a94a51-90f5-4f9b-9118-3c1a85eb8690-kube-api-access-5v86h" pod "67a94a51-90f5-4f9b-9118-3c1a85eb8690" (UID: "67a94a51-90f5-4f9b-9118-3c1a85eb8690"). InnerVolumeSpecName "kube-api-access-5v86h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:38.683559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.683534 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:38.701199 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.701151 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" podStartSLOduration=2.701132832 podStartE2EDuration="2.701132832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:38.698253981 +0000 UTC m=+1326.987910510" watchObservedRunningTime="2026-11-02 20:36:38.701132832 +0000 UTC m=+1326.990789368" Nov 02 20:36:38.747475 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.747441 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:38.760315 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.760282 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 20:36:38.760657 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.760628 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" podUID="714bd47a-f22d-42ac-9692-b6080d740fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 20:36:38.782149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.781437 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a94a51-90f5-4f9b-9118-3c1a85eb8690-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:38.782149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.781477 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a94a51-90f5-4f9b-9118-3c1a85eb8690-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:38.782149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:38.781493 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v86h\" (UniqueName: \"kubernetes.io/projected/67a94a51-90f5-4f9b-9118-3c1a85eb8690-kube-api-access-5v86h\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:39.308796 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:39.308754 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" event={"ID":"67a94a51-90f5-4f9b-9118-3c1a85eb8690","Type":"ContainerDied","Data":"6bd1d5885a9e99b5121c401cedbced1b5982c7c4a4d2e66a5444266d9e8df8bc"} Nov 02 20:36:39.309356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:39.308819 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bd1d5885a9e99b5121c401cedbced1b5982c7c4a4d2e66a5444266d9e8df8bc" Nov 02 20:36:39.309356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:39.308972 4971 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:36:39.309356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:39.309005 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" podUID="f0be871c-832b-4e45-965c-d092b43030eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 20:36:39.309356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:39.309151 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p" Nov 02 20:36:39.707228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:39.707196 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:36:39.707468 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:39.707445 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" podUID="ea994d9c-d955-4efa-9eea-9225a6fee64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 20:36:39.742614 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:39.742493 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 20:36:39.742614 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:39.742555 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" podUID="714bd47a-f22d-42ac-9692-b6080d740fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 20:36:40.071247 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:40.071152 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp"] Nov 02 20:36:40.102105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:40.096647 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b28aad-566wp"] Nov 02 20:36:40.309567 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:40.309522 4971 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:36:40.310234 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:40.309597 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" podUID="f0be871c-832b-4e45-965c-d092b43030eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 20:36:40.697131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:40.694493 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:36:40.697131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:40.694558 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" podUID="ea994d9c-d955-4efa-9eea-9225a6fee64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 20:36:40.744108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:40.739474 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 20:36:40.744108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:40.739531 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" podUID="714bd47a-f22d-42ac-9692-b6080d740fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 20:36:41.085167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:41.083793 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz"] Nov 02 20:36:41.115000 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:41.114963 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94cee0f6-r6mgz"] Nov 02 20:36:41.309424 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:41.309380 4971 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:36:41.309573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:41.309447 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" podUID="f0be871c-832b-4e45-965c-d092b43030eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 20:36:41.686109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:41.685604 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:36:41.686109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:41.685671 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" podUID="ea994d9c-d955-4efa-9eea-9225a6fee64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 20:36:41.742108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:41.741959 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 20:36:41.742108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:41.742016 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" podUID="714bd47a-f22d-42ac-9692-b6080d740fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 20:36:41.908341 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:41.908280 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37" path="/var/lib/kubelet/pods/1c2ed8b8-10c0-4e39-b6bc-1f999a5efc37/volumes" Nov 02 20:36:41.908850 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:41.908835 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83fd427a-ec1c-4eb0-8b0d-73a49c0174e4" path="/var/lib/kubelet/pods/83fd427a-ec1c-4eb0-8b0d-73a49c0174e4/volumes" Nov 02 20:36:42.309165 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:42.309126 4971 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:36:42.309350 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:42.309178 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" podUID="f0be871c-832b-4e45-965c-d092b43030eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 20:36:42.687110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:42.686516 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:36:42.687110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:42.686576 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" podUID="ea994d9c-d955-4efa-9eea-9225a6fee64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 20:36:42.764112 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:42.761304 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 20:36:42.764112 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:42.761362 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" podUID="714bd47a-f22d-42ac-9692-b6080d740fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 20:36:42.939162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:42.937629 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:43.065138 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:43.063497 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4"] Nov 02 20:36:43.117230 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:43.117064 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d64601e-cvsp4"] Nov 02 20:36:43.309114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:43.309017 4971 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:36:43.309114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:43.309073 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" podUID="f0be871c-832b-4e45-965c-d092b43030eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 20:36:43.683282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:43.683248 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:36:43.683486 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:43.683307 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" podUID="ea994d9c-d955-4efa-9eea-9225a6fee64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 20:36:43.742229 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:43.742183 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 20:36:43.742839 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:43.742807 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" podUID="714bd47a-f22d-42ac-9692-b6080d740fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 20:36:43.884588 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:43.884543 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc8a70c-008d-4aba-878c-4f533ccd1930" path="/var/lib/kubelet/pods/7bc8a70c-008d-4aba-878c-4f533ccd1930/volumes" Nov 02 20:36:44.309348 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:44.309310 4971 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 02 20:36:44.309510 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:44.309384 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" podUID="f0be871c-832b-4e45-965c-d092b43030eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 02 20:36:44.683523 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:44.683483 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:36:44.683746 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:44.683562 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" podUID="ea994d9c-d955-4efa-9eea-9225a6fee64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 20:36:45.005898 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:45.005814 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:45.008026 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:45.008004 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:45.027321 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:45.027284 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:36:45.557027 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:45.556969 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:45.557637 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:45.557613 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:45.558888 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:45.558863 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:36:45.930324 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:45.930288 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn"] Nov 02 20:36:45.931029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:45.931009 4971 cpu_manager.go:414] "containerMap: removing container" podUID="67a94a51-90f5-4f9b-9118-3c1a85eb8690" containerName="revoke-keys" Nov 02 20:36:45.931167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:45.931031 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a94a51-90f5-4f9b-9118-3c1a85eb8690" containerName="revoke-keys" Nov 02 20:36:45.931269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:45.931254 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a94a51-90f5-4f9b-9118-3c1a85eb8690" containerName="revoke-keys" Nov 02 20:36:45.935168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:45.935152 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" Nov 02 20:36:45.984964 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:45.984941 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:36:45.985596 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:45.985567 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:36:45.998773 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:45.998736 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn"] Nov 02 20:36:46.024681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.024646 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e45148-2d2hn\" (UID: \"09619f70-5405-41ed-94fd-5f6f5e0fe0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" Nov 02 20:36:46.025800 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.025774 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e45148-2d2hn\" (UID: \"09619f70-5405-41ed-94fd-5f6f5e0fe0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" Nov 02 20:36:46.026036 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.026017 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhggm\" (UniqueName: \"kubernetes.io/projected/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-kube-api-access-mhggm\") pod \"maas-api-revoke-sub-28e45148-2d2hn\" (UID: \"09619f70-5405-41ed-94fd-5f6f5e0fe0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" Nov 02 20:36:46.060782 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.060604 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:46.064925 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.063795 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:46.065279 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.065171 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:36:46.142664 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.142641 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhggm\" (UniqueName: \"kubernetes.io/projected/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-kube-api-access-mhggm\") pod \"maas-api-revoke-sub-28e45148-2d2hn\" (UID: \"09619f70-5405-41ed-94fd-5f6f5e0fe0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" Nov 02 20:36:46.142773 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.142720 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e45148-2d2hn\" (UID: \"09619f70-5405-41ed-94fd-5f6f5e0fe0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" Nov 02 20:36:46.142917 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.142901 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e45148-2d2hn\" (UID: \"09619f70-5405-41ed-94fd-5f6f5e0fe0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" Nov 02 20:36:46.143428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.143407 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e45148-2d2hn\" (UID: \"09619f70-5405-41ed-94fd-5f6f5e0fe0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" Nov 02 20:36:46.145462 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.145441 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e45148-2d2hn\" (UID: \"09619f70-5405-41ed-94fd-5f6f5e0fe0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" Nov 02 20:36:46.202924 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.202868 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhggm\" (UniqueName: \"kubernetes.io/projected/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-kube-api-access-mhggm\") pod \"maas-api-revoke-sub-28e45148-2d2hn\" (UID: \"09619f70-5405-41ed-94fd-5f6f5e0fe0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" Nov 02 20:36:46.219636 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.219615 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv"] Nov 02 20:36:46.224480 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.224443 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" Nov 02 20:36:46.243866 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.243841 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53925b8-a1db-4110-a978-8a32f6f445ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edac9fbd-swqnv\" (UID: \"a53925b8-a1db-4110-a978-8a32f6f445ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" Nov 02 20:36:46.243971 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.243913 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53925b8-a1db-4110-a978-8a32f6f445ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edac9fbd-swqnv\" (UID: \"a53925b8-a1db-4110-a978-8a32f6f445ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" Nov 02 20:36:46.244015 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.244002 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4npcd\" (UniqueName: \"kubernetes.io/projected/a53925b8-a1db-4110-a978-8a32f6f445ed-kube-api-access-4npcd\") pod \"maas-api-revoke-sub-edac9fbd-swqnv\" (UID: \"a53925b8-a1db-4110-a978-8a32f6f445ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" Nov 02 20:36:46.250381 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.250362 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" Nov 02 20:36:46.277979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.277945 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv"] Nov 02 20:36:46.347140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.345466 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53925b8-a1db-4110-a978-8a32f6f445ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edac9fbd-swqnv\" (UID: \"a53925b8-a1db-4110-a978-8a32f6f445ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" Nov 02 20:36:46.347140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.345555 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4npcd\" (UniqueName: \"kubernetes.io/projected/a53925b8-a1db-4110-a978-8a32f6f445ed-kube-api-access-4npcd\") pod \"maas-api-revoke-sub-edac9fbd-swqnv\" (UID: \"a53925b8-a1db-4110-a978-8a32f6f445ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" Nov 02 20:36:46.347140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.345862 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53925b8-a1db-4110-a978-8a32f6f445ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edac9fbd-swqnv\" (UID: \"a53925b8-a1db-4110-a978-8a32f6f445ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" Nov 02 20:36:46.347140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.346724 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53925b8-a1db-4110-a978-8a32f6f445ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edac9fbd-swqnv\" (UID: \"a53925b8-a1db-4110-a978-8a32f6f445ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" Nov 02 20:36:46.349832 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.349807 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53925b8-a1db-4110-a978-8a32f6f445ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edac9fbd-swqnv\" (UID: \"a53925b8-a1db-4110-a978-8a32f6f445ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" Nov 02 20:36:46.417430 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.417390 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4npcd\" (UniqueName: \"kubernetes.io/projected/a53925b8-a1db-4110-a978-8a32f6f445ed-kube-api-access-4npcd\") pod \"maas-api-revoke-sub-edac9fbd-swqnv\" (UID: \"a53925b8-a1db-4110-a978-8a32f6f445ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" Nov 02 20:36:46.540256 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.540191 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" Nov 02 20:36:46.643954 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:36:46.643917 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09619f70_5405_41ed_94fd_5f6f5e0fe0f6.slice/crio-4b148d548b164bf8d7d5a53680dcab9257ec7cd4ee254e228f3c3defb74ea2c1 WatchSource:0}: Error finding container 4b148d548b164bf8d7d5a53680dcab9257ec7cd4ee254e228f3c3defb74ea2c1: Status 404 returned error can't find the container with id 4b148d548b164bf8d7d5a53680dcab9257ec7cd4ee254e228f3c3defb74ea2c1 Nov 02 20:36:46.644333 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.644310 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn"] Nov 02 20:36:46.913645 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:36:46.913595 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53925b8_a1db_4110_a978_8a32f6f445ed.slice/crio-b2367307c48e03f8ac4780b45c25891827667d63500d4a2ffc2f7d6373fdb80d WatchSource:0}: Error finding container b2367307c48e03f8ac4780b45c25891827667d63500d4a2ffc2f7d6373fdb80d: Status 404 returned error can't find the container with id b2367307c48e03f8ac4780b45c25891827667d63500d4a2ffc2f7d6373fdb80d Nov 02 20:36:46.916045 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:46.916021 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv"] Nov 02 20:36:47.455051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:47.455023 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" event={"ID":"09619f70-5405-41ed-94fd-5f6f5e0fe0f6","Type":"ContainerStarted","Data":"421bc6d758629abcc0fc3488118ce570127911f2efb44020a12bbd7cf46adbf1"} Nov 02 20:36:47.455452 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:47.455126 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" event={"ID":"09619f70-5405-41ed-94fd-5f6f5e0fe0f6","Type":"ContainerStarted","Data":"4b148d548b164bf8d7d5a53680dcab9257ec7cd4ee254e228f3c3defb74ea2c1"} Nov 02 20:36:47.456962 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:47.456932 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" event={"ID":"a53925b8-a1db-4110-a978-8a32f6f445ed","Type":"ContainerStarted","Data":"3de6b3bcd773ecc5c88c7ec8d9d2269c36e9934777ac1187e69206f42b9e0841"} Nov 02 20:36:47.457068 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:47.457000 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" event={"ID":"a53925b8-a1db-4110-a978-8a32f6f445ed","Type":"ContainerStarted","Data":"b2367307c48e03f8ac4780b45c25891827667d63500d4a2ffc2f7d6373fdb80d"} Nov 02 20:36:47.564196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:47.564140 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" podStartSLOduration=2.564128617 podStartE2EDuration="2.564128617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:47.563659611 +0000 UTC m=+1335.853316139" watchObservedRunningTime="2026-11-02 20:36:47.564128617 +0000 UTC m=+1335.853785154" Nov 02 20:36:47.673968 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:47.673917 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" podStartSLOduration=1.6739030449999999 podStartE2EDuration="1.673903045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:47.671968649 +0000 UTC m=+1335.961625178" watchObservedRunningTime="2026-11-02 20:36:47.673903045 +0000 UTC m=+1335.963559596" Nov 02 20:36:47.923551 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:47.923509 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:48.466328 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:48.466297 4971 generic.go:356] "Generic (PLEG): container finished" podID="a53925b8-a1db-4110-a978-8a32f6f445ed" containerID="3de6b3bcd773ecc5c88c7ec8d9d2269c36e9934777ac1187e69206f42b9e0841" exitCode=0 Nov 02 20:36:48.466757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:48.466397 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" event={"ID":"a53925b8-a1db-4110-a978-8a32f6f445ed","Type":"ContainerDied","Data":"3de6b3bcd773ecc5c88c7ec8d9d2269c36e9934777ac1187e69206f42b9e0841"} Nov 02 20:36:48.468076 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:48.468054 4971 generic.go:356] "Generic (PLEG): container finished" podID="09619f70-5405-41ed-94fd-5f6f5e0fe0f6" containerID="421bc6d758629abcc0fc3488118ce570127911f2efb44020a12bbd7cf46adbf1" exitCode=0 Nov 02 20:36:48.468221 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:48.468119 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" event={"ID":"09619f70-5405-41ed-94fd-5f6f5e0fe0f6","Type":"ContainerDied","Data":"421bc6d758629abcc0fc3488118ce570127911f2efb44020a12bbd7cf46adbf1"} Nov 02 20:36:49.679530 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.679504 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" Nov 02 20:36:49.700215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.700196 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" Nov 02 20:36:49.790511 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.790435 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-maas-api-openshift-ca\") pod \"09619f70-5405-41ed-94fd-5f6f5e0fe0f6\" (UID: \"09619f70-5405-41ed-94fd-5f6f5e0fe0f6\") " Nov 02 20:36:49.790511 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.790491 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-maas-api-xks-ca\") pod \"09619f70-5405-41ed-94fd-5f6f5e0fe0f6\" (UID: \"09619f70-5405-41ed-94fd-5f6f5e0fe0f6\") " Nov 02 20:36:49.790691 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.790520 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a53925b8-a1db-4110-a978-8a32f6f445ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53925b8-a1db-4110-a978-8a32f6f445ed-maas-api-xks-ca\") pod \"a53925b8-a1db-4110-a978-8a32f6f445ed\" (UID: \"a53925b8-a1db-4110-a978-8a32f6f445ed\") " Nov 02 20:36:49.790691 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.790576 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53925b8-a1db-4110-a978-8a32f6f445ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53925b8-a1db-4110-a978-8a32f6f445ed-maas-api-openshift-ca\") pod \"a53925b8-a1db-4110-a978-8a32f6f445ed\" (UID: \"a53925b8-a1db-4110-a978-8a32f6f445ed\") " Nov 02 20:36:49.790691 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.790634 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53925b8-a1db-4110-a978-8a32f6f445ed-kube-api-access-4npcd\" (UniqueName: \"kubernetes.io/projected/a53925b8-a1db-4110-a978-8a32f6f445ed-kube-api-access-4npcd\") pod \"a53925b8-a1db-4110-a978-8a32f6f445ed\" (UID: \"a53925b8-a1db-4110-a978-8a32f6f445ed\") " Nov 02 20:36:49.790839 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.790774 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-kube-api-access-mhggm\" (UniqueName: \"kubernetes.io/projected/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-kube-api-access-mhggm\") pod \"09619f70-5405-41ed-94fd-5f6f5e0fe0f6\" (UID: \"09619f70-5405-41ed-94fd-5f6f5e0fe0f6\") " Nov 02 20:36:49.791528 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.791108 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-maas-api-openshift-ca" pod "09619f70-5405-41ed-94fd-5f6f5e0fe0f6" (UID: "09619f70-5405-41ed-94fd-5f6f5e0fe0f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:49.791528 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.791485 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:49.792014 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.791974 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53925b8-a1db-4110-a978-8a32f6f445ed-maas-api-openshift-ca" pod "a53925b8-a1db-4110-a978-8a32f6f445ed" (UID: "a53925b8-a1db-4110-a978-8a32f6f445ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:49.793305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.793274 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a53925b8-a1db-4110-a978-8a32f6f445ed-maas-api-xks-ca" pod "a53925b8-a1db-4110-a978-8a32f6f445ed" (UID: "a53925b8-a1db-4110-a978-8a32f6f445ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:49.793686 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.793655 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-maas-api-xks-ca" pod "09619f70-5405-41ed-94fd-5f6f5e0fe0f6" (UID: "09619f70-5405-41ed-94fd-5f6f5e0fe0f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:49.794025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.793991 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-kube-api-access-mhggm" pod "09619f70-5405-41ed-94fd-5f6f5e0fe0f6" (UID: "09619f70-5405-41ed-94fd-5f6f5e0fe0f6"). InnerVolumeSpecName "kube-api-access-mhggm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:49.795232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.795203 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53925b8-a1db-4110-a978-8a32f6f445ed-kube-api-access-4npcd" pod "a53925b8-a1db-4110-a978-8a32f6f445ed" (UID: "a53925b8-a1db-4110-a978-8a32f6f445ed"). InnerVolumeSpecName "kube-api-access-4npcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:49.892505 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.892479 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:49.892635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.892512 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53925b8-a1db-4110-a978-8a32f6f445ed-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:49.892635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.892531 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53925b8-a1db-4110-a978-8a32f6f445ed-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:49.892635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.892549 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4npcd\" (UniqueName: \"kubernetes.io/projected/a53925b8-a1db-4110-a978-8a32f6f445ed-kube-api-access-4npcd\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:49.892635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:49.892566 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhggm\" (UniqueName: \"kubernetes.io/projected/09619f70-5405-41ed-94fd-5f6f5e0fe0f6-kube-api-access-mhggm\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:50.495699 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.495673 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" Nov 02 20:36:50.495870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.495718 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn" event={"ID":"09619f70-5405-41ed-94fd-5f6f5e0fe0f6","Type":"ContainerDied","Data":"4b148d548b164bf8d7d5a53680dcab9257ec7cd4ee254e228f3c3defb74ea2c1"} Nov 02 20:36:50.495870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.495768 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b148d548b164bf8d7d5a53680dcab9257ec7cd4ee254e228f3c3defb74ea2c1" Nov 02 20:36:50.497363 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.497343 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" Nov 02 20:36:50.497464 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.497369 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv" event={"ID":"a53925b8-a1db-4110-a978-8a32f6f445ed","Type":"ContainerDied","Data":"b2367307c48e03f8ac4780b45c25891827667d63500d4a2ffc2f7d6373fdb80d"} Nov 02 20:36:50.497464 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.497402 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2367307c48e03f8ac4780b45c25891827667d63500d4a2ffc2f7d6373fdb80d" Nov 02 20:36:50.672270 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.672232 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw"] Nov 02 20:36:50.673367 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.673347 4971 cpu_manager.go:414] "containerMap: removing container" podUID="09619f70-5405-41ed-94fd-5f6f5e0fe0f6" containerName="revoke-keys" Nov 02 20:36:50.673416 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.673371 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09619f70-5405-41ed-94fd-5f6f5e0fe0f6" containerName="revoke-keys" Nov 02 20:36:50.673416 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.673394 4971 cpu_manager.go:414] "containerMap: removing container" podUID="a53925b8-a1db-4110-a978-8a32f6f445ed" containerName="revoke-keys" Nov 02 20:36:50.673416 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.673402 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53925b8-a1db-4110-a978-8a32f6f445ed" containerName="revoke-keys" Nov 02 20:36:50.673574 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.673564 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53925b8-a1db-4110-a978-8a32f6f445ed" containerName="revoke-keys" Nov 02 20:36:50.673613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.673596 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="09619f70-5405-41ed-94fd-5f6f5e0fe0f6" containerName="revoke-keys" Nov 02 20:36:50.678118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.678099 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw" Nov 02 20:36:50.728967 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.727669 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:36:50.728967 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.727714 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:36:50.730373 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.730348 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw"] Nov 02 20:36:50.804670 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.804597 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0981e22-b642-418b-9a57-482a92589ec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea955427-5fhsw\" (UID: \"f0981e22-b642-418b-9a57-482a92589ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw" Nov 02 20:36:50.804670 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.804650 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0981e22-b642-418b-9a57-482a92589ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea955427-5fhsw\" (UID: \"f0981e22-b642-418b-9a57-482a92589ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw" Nov 02 20:36:50.804863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.804748 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9vh4\" (UniqueName: \"kubernetes.io/projected/f0981e22-b642-418b-9a57-482a92589ec4-kube-api-access-t9vh4\") pod \"maas-api-revoke-sub-ea955427-5fhsw\" (UID: \"f0981e22-b642-418b-9a57-482a92589ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw" Nov 02 20:36:50.905582 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.905553 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9vh4\" (UniqueName: \"kubernetes.io/projected/f0981e22-b642-418b-9a57-482a92589ec4-kube-api-access-t9vh4\") pod \"maas-api-revoke-sub-ea955427-5fhsw\" (UID: \"f0981e22-b642-418b-9a57-482a92589ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw" Nov 02 20:36:50.905789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.905767 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0981e22-b642-418b-9a57-482a92589ec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea955427-5fhsw\" (UID: \"f0981e22-b642-418b-9a57-482a92589ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw" Nov 02 20:36:50.905918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.905829 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0981e22-b642-418b-9a57-482a92589ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea955427-5fhsw\" (UID: \"f0981e22-b642-418b-9a57-482a92589ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw" Nov 02 20:36:50.906520 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.906493 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0981e22-b642-418b-9a57-482a92589ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea955427-5fhsw\" (UID: \"f0981e22-b642-418b-9a57-482a92589ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw" Nov 02 20:36:50.908820 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.908799 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0981e22-b642-418b-9a57-482a92589ec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea955427-5fhsw\" (UID: \"f0981e22-b642-418b-9a57-482a92589ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw" Nov 02 20:36:50.964165 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.964136 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9vh4\" (UniqueName: \"kubernetes.io/projected/f0981e22-b642-418b-9a57-482a92589ec4-kube-api-access-t9vh4\") pod \"maas-api-revoke-sub-ea955427-5fhsw\" (UID: \"f0981e22-b642-418b-9a57-482a92589ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw" Nov 02 20:36:50.994625 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:50.994601 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw" Nov 02 20:36:51.050408 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:51.050373 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb"] Nov 02 20:36:51.060506 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:51.057586 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67dbdfb1-skpfb"] Nov 02 20:36:51.382177 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:36:51.382147 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0981e22_b642_418b_9a57_482a92589ec4.slice/crio-0bf0b6309ad58fea31620aaeb427298eb8145e6831ddb8c5d1913f023e353ffa WatchSource:0}: Error finding container 0bf0b6309ad58fea31620aaeb427298eb8145e6831ddb8c5d1913f023e353ffa: Status 404 returned error can't find the container with id 0bf0b6309ad58fea31620aaeb427298eb8145e6831ddb8c5d1913f023e353ffa Nov 02 20:36:51.382495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:51.382475 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw"] Nov 02 20:36:51.508417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:51.508385 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw" event={"ID":"f0981e22-b642-418b-9a57-482a92589ec4","Type":"ContainerStarted","Data":"0bf0b6309ad58fea31620aaeb427298eb8145e6831ddb8c5d1913f023e353ffa"} Nov 02 20:36:51.867698 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:51.867667 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52a6ba6-42e7-4cb8-b04d-9f50939de2a5" path="/var/lib/kubelet/pods/d52a6ba6-42e7-4cb8-b04d-9f50939de2a5/volumes" Nov 02 20:36:51.901813 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:51.901784 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc"] Nov 02 20:36:51.908045 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:51.908015 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc" Nov 02 20:36:51.960425 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:51.960393 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc"] Nov 02 20:36:52.021639 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:52.021569 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vpd5\" (UniqueName: \"kubernetes.io/projected/d95853df-8304-4539-8e10-60a57f2ab497-kube-api-access-5vpd5\") pod \"maas-api-revoke-sub-ba4233da-lh6qc\" (UID: \"d95853df-8304-4539-8e10-60a57f2ab497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc" Nov 02 20:36:52.021639 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:52.021634 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95853df-8304-4539-8e10-60a57f2ab497-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba4233da-lh6qc\" (UID: \"d95853df-8304-4539-8e10-60a57f2ab497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc" Nov 02 20:36:52.021835 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:52.021708 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95853df-8304-4539-8e10-60a57f2ab497-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba4233da-lh6qc\" (UID: \"d95853df-8304-4539-8e10-60a57f2ab497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc" Nov 02 20:36:52.047180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:52.047155 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz"] Nov 02 20:36:52.067218 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:52.067189 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-288cc6ec-gpxwz"] Nov 02 20:36:52.122964 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:52.122935 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vpd5\" (UniqueName: \"kubernetes.io/projected/d95853df-8304-4539-8e10-60a57f2ab497-kube-api-access-5vpd5\") pod \"maas-api-revoke-sub-ba4233da-lh6qc\" (UID: \"d95853df-8304-4539-8e10-60a57f2ab497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc" Nov 02 20:36:52.123143 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:52.123034 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95853df-8304-4539-8e10-60a57f2ab497-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba4233da-lh6qc\" (UID: \"d95853df-8304-4539-8e10-60a57f2ab497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc" Nov 02 20:36:52.123143 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:52.123096 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95853df-8304-4539-8e10-60a57f2ab497-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba4233da-lh6qc\" (UID: \"d95853df-8304-4539-8e10-60a57f2ab497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc" Nov 02 20:36:52.123723 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:52.123701 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95853df-8304-4539-8e10-60a57f2ab497-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba4233da-lh6qc\" (UID: \"d95853df-8304-4539-8e10-60a57f2ab497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc" Nov 02 20:36:52.125776 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:52.125750 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95853df-8304-4539-8e10-60a57f2ab497-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba4233da-lh6qc\" (UID: \"d95853df-8304-4539-8e10-60a57f2ab497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc" Nov 02 20:36:52.180971 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:52.180947 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vpd5\" (UniqueName: \"kubernetes.io/projected/d95853df-8304-4539-8e10-60a57f2ab497-kube-api-access-5vpd5\") pod \"maas-api-revoke-sub-ba4233da-lh6qc\" (UID: \"d95853df-8304-4539-8e10-60a57f2ab497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc" Nov 02 20:36:52.229882 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:52.229862 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc" Nov 02 20:36:52.518274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:52.518237 4971 generic.go:356] "Generic (PLEG): container finished" podID="f0981e22-b642-418b-9a57-482a92589ec4" containerID="cf161819604cf28ffff9a606c918eeff13b147a97a06eb58286f0e968b2a4a14" exitCode=0 Nov 02 20:36:52.518274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:52.518275 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw" event={"ID":"f0981e22-b642-418b-9a57-482a92589ec4","Type":"ContainerDied","Data":"cf161819604cf28ffff9a606c918eeff13b147a97a06eb58286f0e968b2a4a14"} Nov 02 20:36:52.605790 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:36:52.605764 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95853df_8304_4539_8e10_60a57f2ab497.slice/crio-e77d5b61aa58daa29bbe48c0c4bc3be3dcd739c699316e706aac35cb59538a59 WatchSource:0}: Error finding container e77d5b61aa58daa29bbe48c0c4bc3be3dcd739c699316e706aac35cb59538a59: Status 404 returned error can't find the container with id e77d5b61aa58daa29bbe48c0c4bc3be3dcd739c699316e706aac35cb59538a59 Nov 02 20:36:52.606963 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:52.606941 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc"] Nov 02 20:36:52.808704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:52.808667 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:53.528437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:53.528402 4971 generic.go:356] "Generic (PLEG): container finished" podID="d95853df-8304-4539-8e10-60a57f2ab497" containerID="04c831b82de2675d821b6146fa69b57e3cfca0ab14563e03f6129e623d555859" exitCode=0 Nov 02 20:36:53.528850 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:53.528533 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc" event={"ID":"d95853df-8304-4539-8e10-60a57f2ab497","Type":"ContainerDied","Data":"04c831b82de2675d821b6146fa69b57e3cfca0ab14563e03f6129e623d555859"} Nov 02 20:36:53.528850 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:53.528592 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc" event={"ID":"d95853df-8304-4539-8e10-60a57f2ab497","Type":"ContainerStarted","Data":"e77d5b61aa58daa29bbe48c0c4bc3be3dcd739c699316e706aac35cb59538a59"} Nov 02 20:36:53.711343 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:53.711318 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw" Nov 02 20:36:53.846329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:53.846293 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0981e22-b642-418b-9a57-482a92589ec4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0981e22-b642-418b-9a57-482a92589ec4-maas-api-openshift-ca\") pod \"f0981e22-b642-418b-9a57-482a92589ec4\" (UID: \"f0981e22-b642-418b-9a57-482a92589ec4\") " Nov 02 20:36:53.846497 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:53.846334 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0981e22-b642-418b-9a57-482a92589ec4-kube-api-access-t9vh4\" (UniqueName: \"kubernetes.io/projected/f0981e22-b642-418b-9a57-482a92589ec4-kube-api-access-t9vh4\") pod \"f0981e22-b642-418b-9a57-482a92589ec4\" (UID: \"f0981e22-b642-418b-9a57-482a92589ec4\") " Nov 02 20:36:53.846497 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:53.846419 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0981e22-b642-418b-9a57-482a92589ec4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0981e22-b642-418b-9a57-482a92589ec4-maas-api-xks-ca\") pod \"f0981e22-b642-418b-9a57-482a92589ec4\" (UID: \"f0981e22-b642-418b-9a57-482a92589ec4\") " Nov 02 20:36:53.846660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:53.846634 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0981e22-b642-418b-9a57-482a92589ec4-maas-api-openshift-ca" pod "f0981e22-b642-418b-9a57-482a92589ec4" (UID: "f0981e22-b642-418b-9a57-482a92589ec4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:53.846941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:53.846923 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0981e22-b642-418b-9a57-482a92589ec4-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:53.848686 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:53.848660 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0981e22-b642-418b-9a57-482a92589ec4-maas-api-xks-ca" pod "f0981e22-b642-418b-9a57-482a92589ec4" (UID: "f0981e22-b642-418b-9a57-482a92589ec4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:53.848686 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:53.848674 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0981e22-b642-418b-9a57-482a92589ec4-kube-api-access-t9vh4" pod "f0981e22-b642-418b-9a57-482a92589ec4" (UID: "f0981e22-b642-418b-9a57-482a92589ec4"). InnerVolumeSpecName "kube-api-access-t9vh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:53.874637 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:53.874604 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa2bd90-0708-4c95-aa5d-9df2b3d1965e" path="/var/lib/kubelet/pods/cfa2bd90-0708-4c95-aa5d-9df2b3d1965e/volumes" Nov 02 20:36:53.947840 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:53.947816 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9vh4\" (UniqueName: \"kubernetes.io/projected/f0981e22-b642-418b-9a57-482a92589ec4-kube-api-access-t9vh4\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:53.947840 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:53.947841 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0981e22-b642-418b-9a57-482a92589ec4-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:54.050854 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:54.050826 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw"] Nov 02 20:36:54.055636 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:54.055613 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-419a3148-kx9dw"] Nov 02 20:36:54.537950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:54.537917 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw" Nov 02 20:36:54.537950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:54.537938 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw" event={"ID":"f0981e22-b642-418b-9a57-482a92589ec4","Type":"ContainerDied","Data":"0bf0b6309ad58fea31620aaeb427298eb8145e6831ddb8c5d1913f023e353ffa"} Nov 02 20:36:54.538426 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:54.537991 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bf0b6309ad58fea31620aaeb427298eb8145e6831ddb8c5d1913f023e353ffa" Nov 02 20:36:54.719488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:54.719464 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc" Nov 02 20:36:54.861114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:54.861069 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95853df-8304-4539-8e10-60a57f2ab497-kube-api-access-5vpd5\" (UniqueName: \"kubernetes.io/projected/d95853df-8304-4539-8e10-60a57f2ab497-kube-api-access-5vpd5\") pod \"d95853df-8304-4539-8e10-60a57f2ab497\" (UID: \"d95853df-8304-4539-8e10-60a57f2ab497\") " Nov 02 20:36:54.861299 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:54.861197 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95853df-8304-4539-8e10-60a57f2ab497-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95853df-8304-4539-8e10-60a57f2ab497-maas-api-openshift-ca\") pod \"d95853df-8304-4539-8e10-60a57f2ab497\" (UID: \"d95853df-8304-4539-8e10-60a57f2ab497\") " Nov 02 20:36:54.861431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:54.861409 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d95853df-8304-4539-8e10-60a57f2ab497-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95853df-8304-4539-8e10-60a57f2ab497-maas-api-xks-ca\") pod \"d95853df-8304-4539-8e10-60a57f2ab497\" (UID: \"d95853df-8304-4539-8e10-60a57f2ab497\") " Nov 02 20:36:54.861606 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:54.861565 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95853df-8304-4539-8e10-60a57f2ab497-maas-api-openshift-ca" pod "d95853df-8304-4539-8e10-60a57f2ab497" (UID: "d95853df-8304-4539-8e10-60a57f2ab497"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:54.862130 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:54.862108 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95853df-8304-4539-8e10-60a57f2ab497-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:54.863596 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:54.863571 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95853df-8304-4539-8e10-60a57f2ab497-kube-api-access-5vpd5" pod "d95853df-8304-4539-8e10-60a57f2ab497" (UID: "d95853df-8304-4539-8e10-60a57f2ab497"). InnerVolumeSpecName "kube-api-access-5vpd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:54.865031 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:54.865007 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95853df-8304-4539-8e10-60a57f2ab497-maas-api-xks-ca" pod "d95853df-8304-4539-8e10-60a57f2ab497" (UID: "d95853df-8304-4539-8e10-60a57f2ab497"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:54.963386 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:54.963364 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95853df-8304-4539-8e10-60a57f2ab497-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:54.963386 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:54.963387 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vpd5\" (UniqueName: \"kubernetes.io/projected/d95853df-8304-4539-8e10-60a57f2ab497-kube-api-access-5vpd5\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:55.144942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.144867 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9"] Nov 02 20:36:55.145964 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.145943 4971 cpu_manager.go:414] "containerMap: removing container" podUID="f0981e22-b642-418b-9a57-482a92589ec4" containerName="revoke-keys" Nov 02 20:36:55.146047 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.145967 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0981e22-b642-418b-9a57-482a92589ec4" containerName="revoke-keys" Nov 02 20:36:55.146047 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.146022 4971 cpu_manager.go:414] "containerMap: removing container" podUID="d95853df-8304-4539-8e10-60a57f2ab497" containerName="revoke-keys" Nov 02 20:36:55.146047 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.146032 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95853df-8304-4539-8e10-60a57f2ab497" containerName="revoke-keys" Nov 02 20:36:55.146227 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.146214 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95853df-8304-4539-8e10-60a57f2ab497" containerName="revoke-keys" Nov 02 20:36:55.146267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.146244 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0981e22-b642-418b-9a57-482a92589ec4" containerName="revoke-keys" Nov 02 20:36:55.156018 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.155986 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" Nov 02 20:36:55.203381 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.203348 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9"] Nov 02 20:36:55.267791 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.267757 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19af21aa-7hvp9\" (UID: \"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" Nov 02 20:36:55.267947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.267898 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njzj\" (UniqueName: \"kubernetes.io/projected/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-kube-api-access-9njzj\") pod \"maas-api-revoke-sub-19af21aa-7hvp9\" (UID: \"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" Nov 02 20:36:55.268023 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.267960 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19af21aa-7hvp9\" (UID: \"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" Nov 02 20:36:55.369344 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.369314 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19af21aa-7hvp9\" (UID: \"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" Nov 02 20:36:55.369484 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.369410 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njzj\" (UniqueName: \"kubernetes.io/projected/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-kube-api-access-9njzj\") pod \"maas-api-revoke-sub-19af21aa-7hvp9\" (UID: \"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" Nov 02 20:36:55.369484 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.369462 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19af21aa-7hvp9\" (UID: \"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" Nov 02 20:36:55.370284 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.370202 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19af21aa-7hvp9\" (UID: \"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" Nov 02 20:36:55.372030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.372004 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19af21aa-7hvp9\" (UID: \"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" Nov 02 20:36:55.429865 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.429808 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njzj\" (UniqueName: \"kubernetes.io/projected/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-kube-api-access-9njzj\") pod \"maas-api-revoke-sub-19af21aa-7hvp9\" (UID: \"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" Nov 02 20:36:55.482103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.482034 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" Nov 02 20:36:55.575104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.574869 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl"] Nov 02 20:36:55.600104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.596886 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" Nov 02 20:36:55.612106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.600599 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc" event={"ID":"d95853df-8304-4539-8e10-60a57f2ab497","Type":"ContainerDied","Data":"e77d5b61aa58daa29bbe48c0c4bc3be3dcd739c699316e706aac35cb59538a59"} Nov 02 20:36:55.612106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.600652 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e77d5b61aa58daa29bbe48c0c4bc3be3dcd739c699316e706aac35cb59538a59" Nov 02 20:36:55.612106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.600713 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc" Nov 02 20:36:55.647106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.644777 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl"] Nov 02 20:36:55.686785 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.686703 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf8j9\" (UniqueName: \"kubernetes.io/projected/3e3c0362-fdcd-4801-9634-657f6ad5dcae-kube-api-access-tf8j9\") pod \"maas-api-revoke-sub-28b84cc4-nfvnl\" (UID: \"3e3c0362-fdcd-4801-9634-657f6ad5dcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" Nov 02 20:36:55.687174 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.687142 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e3c0362-fdcd-4801-9634-657f6ad5dcae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b84cc4-nfvnl\" (UID: \"3e3c0362-fdcd-4801-9634-657f6ad5dcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" Nov 02 20:36:55.687697 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.687676 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e3c0362-fdcd-4801-9634-657f6ad5dcae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b84cc4-nfvnl\" (UID: \"3e3c0362-fdcd-4801-9634-657f6ad5dcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" Nov 02 20:36:55.853114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.839317 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e3c0362-fdcd-4801-9634-657f6ad5dcae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b84cc4-nfvnl\" (UID: \"3e3c0362-fdcd-4801-9634-657f6ad5dcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" Nov 02 20:36:55.853114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.839424 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf8j9\" (UniqueName: \"kubernetes.io/projected/3e3c0362-fdcd-4801-9634-657f6ad5dcae-kube-api-access-tf8j9\") pod \"maas-api-revoke-sub-28b84cc4-nfvnl\" (UID: \"3e3c0362-fdcd-4801-9634-657f6ad5dcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" Nov 02 20:36:55.853114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.839487 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e3c0362-fdcd-4801-9634-657f6ad5dcae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b84cc4-nfvnl\" (UID: \"3e3c0362-fdcd-4801-9634-657f6ad5dcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" Nov 02 20:36:55.853114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.842417 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e3c0362-fdcd-4801-9634-657f6ad5dcae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b84cc4-nfvnl\" (UID: \"3e3c0362-fdcd-4801-9634-657f6ad5dcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" Nov 02 20:36:55.902204 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.902161 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e3c0362-fdcd-4801-9634-657f6ad5dcae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b84cc4-nfvnl\" (UID: \"3e3c0362-fdcd-4801-9634-657f6ad5dcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" Nov 02 20:36:55.923161 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.913158 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf8j9\" (UniqueName: \"kubernetes.io/projected/3e3c0362-fdcd-4801-9634-657f6ad5dcae-kube-api-access-tf8j9\") pod \"maas-api-revoke-sub-28b84cc4-nfvnl\" (UID: \"3e3c0362-fdcd-4801-9634-657f6ad5dcae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" Nov 02 20:36:55.944545 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.939599 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7baadcc3-0541-424e-9bc1-c28eb755787a" path="/var/lib/kubelet/pods/7baadcc3-0541-424e-9bc1-c28eb755787a/volumes" Nov 02 20:36:55.944545 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:55.942558 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" Nov 02 20:36:56.036426 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:56.035412 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9"] Nov 02 20:36:57.018172 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:57.003049 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" event={"ID":"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1","Type":"ContainerStarted","Data":"aeba35ed8dc97b2b9fcf4e148a2d94e66ee7eb23bcf1ec7b57087190970cc7ad"} Nov 02 20:36:57.284105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:57.283287 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:57.313109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:57.310769 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl"] Nov 02 20:36:57.386447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:57.386349 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 02 20:36:57.724239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:57.724172 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:57.892206 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:57.872894 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:57.941185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:57.902153 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:57.986487 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:57.952628 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:58.051272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.050145 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:58.167905 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.133073 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded" Nov 02 20:36:58.167905 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.155575 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 02 20:36:58.192809 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.155851 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 20:36:58.231107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.227171 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 02 20:36:58.292927 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.289100 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 20:36:58.292927 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.290313 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:36:58.330963 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.330885 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" event={"ID":"3e3c0362-fdcd-4801-9634-657f6ad5dcae","Type":"ContainerStarted","Data":"77baef90a5d3065767d25868dc1fac89b25898f1f1e754b471c07a147432f7bd"} Nov 02 20:36:58.334573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.333646 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" event={"ID":"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1","Type":"ContainerStarted","Data":"89fa4906eaaf3e2a42089aab00b56fb8b8072d74a0357eb47e25b2f34379f5dd"} Nov 02 20:36:58.375107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.363098 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Nov 02 20:36:58.410218 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.410167 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:58.436109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.410806 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:36:58.708106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.707264 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded" Nov 02 20:36:58.786459 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.786404 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:58.800613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.789297 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:58.800613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.790178 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:36:58.887113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.884238 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:58.887113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.884792 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:36:58.909011 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.908779 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:36:58.913542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.912875 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:58.914421 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.914402 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:36:58.928149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.919220 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" podStartSLOduration=3.919203211 podStartE2EDuration="3.919203211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:58.908126848 +0000 UTC m=+1347.197783559" watchObservedRunningTime="2026-11-02 20:36:58.919203211 +0000 UTC m=+1347.208859743" Nov 02 20:36:58.928344 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.928326 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:36:58.929203 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.929186 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:36:58.939534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.939518 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:36:58.949569 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:58.949553 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:36:59.300359 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:59.300323 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:36:59.362515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:59.362477 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" event={"ID":"3e3c0362-fdcd-4801-9634-657f6ad5dcae","Type":"ContainerStarted","Data":"f498f98d1d843f8c36327cafff6fab0d705b50d8da57c668b54b17c381849fcd"} Nov 02 20:36:59.364457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:59.364425 4971 generic.go:356] "Generic (PLEG): container finished" podID="39e0a7df-c53e-4fbd-8b6e-90d2adb023c1" containerID="89fa4906eaaf3e2a42089aab00b56fb8b8072d74a0357eb47e25b2f34379f5dd" exitCode=0 Nov 02 20:36:59.364607 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:59.364548 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" event={"ID":"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1","Type":"ContainerDied","Data":"89fa4906eaaf3e2a42089aab00b56fb8b8072d74a0357eb47e25b2f34379f5dd"} Nov 02 20:36:59.803842 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:36:59.803789 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" podStartSLOduration=4.80377317 podStartE2EDuration="4.80377317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:59.690544215 +0000 UTC m=+1347.980200744" watchObservedRunningTime="2026-11-02 20:36:59.80377317 +0000 UTC m=+1348.093429698" Nov 02 20:37:00.052465 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:00.052430 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2"] Nov 02 20:37:00.060462 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:00.060388 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7d2aa81-c64f2"] Nov 02 20:37:00.375768 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:00.375734 4971 generic.go:356] "Generic (PLEG): container finished" podID="3e3c0362-fdcd-4801-9634-657f6ad5dcae" containerID="f498f98d1d843f8c36327cafff6fab0d705b50d8da57c668b54b17c381849fcd" exitCode=0 Nov 02 20:37:00.376280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:00.375814 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" event={"ID":"3e3c0362-fdcd-4801-9634-657f6ad5dcae","Type":"ContainerDied","Data":"f498f98d1d843f8c36327cafff6fab0d705b50d8da57c668b54b17c381849fcd"} Nov 02 20:37:00.587936 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:00.587906 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" Nov 02 20:37:00.645221 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:00.645138 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-kube-api-access-9njzj\" (UniqueName: \"kubernetes.io/projected/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-kube-api-access-9njzj\") pod \"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1\" (UID: \"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1\") " Nov 02 20:37:00.651041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:00.650868 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-maas-api-xks-ca\") pod \"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1\" (UID: \"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1\") " Nov 02 20:37:00.651598 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:00.651568 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-maas-api-openshift-ca" pod "39e0a7df-c53e-4fbd-8b6e-90d2adb023c1" (UID: "39e0a7df-c53e-4fbd-8b6e-90d2adb023c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:00.652192 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:00.652168 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-kube-api-access-9njzj" pod "39e0a7df-c53e-4fbd-8b6e-90d2adb023c1" (UID: "39e0a7df-c53e-4fbd-8b6e-90d2adb023c1"). InnerVolumeSpecName "kube-api-access-9njzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:00.652446 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:00.651287 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-maas-api-openshift-ca\") pod \"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1\" (UID: \"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1\") " Nov 02 20:37:00.653188 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:00.653168 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:00.653285 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:00.653195 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9njzj\" (UniqueName: \"kubernetes.io/projected/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-kube-api-access-9njzj\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:00.653285 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:00.653202 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-maas-api-xks-ca" pod "39e0a7df-c53e-4fbd-8b6e-90d2adb023c1" (UID: "39e0a7df-c53e-4fbd-8b6e-90d2adb023c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:00.754524 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:00.754498 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:01.386019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:01.385993 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" Nov 02 20:37:01.386479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:01.386035 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9" event={"ID":"39e0a7df-c53e-4fbd-8b6e-90d2adb023c1","Type":"ContainerDied","Data":"aeba35ed8dc97b2b9fcf4e148a2d94e66ee7eb23bcf1ec7b57087190970cc7ad"} Nov 02 20:37:01.386479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:01.386112 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aeba35ed8dc97b2b9fcf4e148a2d94e66ee7eb23bcf1ec7b57087190970cc7ad" Nov 02 20:37:01.551434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:01.551402 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" Nov 02 20:37:01.666336 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:01.666260 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e3c0362-fdcd-4801-9634-657f6ad5dcae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e3c0362-fdcd-4801-9634-657f6ad5dcae-maas-api-xks-ca\") pod \"3e3c0362-fdcd-4801-9634-657f6ad5dcae\" (UID: \"3e3c0362-fdcd-4801-9634-657f6ad5dcae\") " Nov 02 20:37:01.666336 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:01.666300 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e3c0362-fdcd-4801-9634-657f6ad5dcae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e3c0362-fdcd-4801-9634-657f6ad5dcae-maas-api-openshift-ca\") pod \"3e3c0362-fdcd-4801-9634-657f6ad5dcae\" (UID: \"3e3c0362-fdcd-4801-9634-657f6ad5dcae\") " Nov 02 20:37:01.666336 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:01.666324 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e3c0362-fdcd-4801-9634-657f6ad5dcae-kube-api-access-tf8j9\" (UniqueName: \"kubernetes.io/projected/3e3c0362-fdcd-4801-9634-657f6ad5dcae-kube-api-access-tf8j9\") pod \"3e3c0362-fdcd-4801-9634-657f6ad5dcae\" (UID: \"3e3c0362-fdcd-4801-9634-657f6ad5dcae\") " Nov 02 20:37:01.666716 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:01.666689 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e3c0362-fdcd-4801-9634-657f6ad5dcae-maas-api-openshift-ca" pod "3e3c0362-fdcd-4801-9634-657f6ad5dcae" (UID: "3e3c0362-fdcd-4801-9634-657f6ad5dcae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:01.667058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:01.667025 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e3c0362-fdcd-4801-9634-657f6ad5dcae-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:01.668654 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:01.668626 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e3c0362-fdcd-4801-9634-657f6ad5dcae-maas-api-xks-ca" pod "3e3c0362-fdcd-4801-9634-657f6ad5dcae" (UID: "3e3c0362-fdcd-4801-9634-657f6ad5dcae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:01.668754 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:01.668732 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3c0362-fdcd-4801-9634-657f6ad5dcae-kube-api-access-tf8j9" pod "3e3c0362-fdcd-4801-9634-657f6ad5dcae" (UID: "3e3c0362-fdcd-4801-9634-657f6ad5dcae"). InnerVolumeSpecName "kube-api-access-tf8j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:01.768574 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:01.768547 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e3c0362-fdcd-4801-9634-657f6ad5dcae-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:01.768574 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:01.768572 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf8j9\" (UniqueName: \"kubernetes.io/projected/3e3c0362-fdcd-4801-9634-657f6ad5dcae-kube-api-access-tf8j9\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:01.869164 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:01.869132 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6e28c13-f0e4-4ebe-a8f7-192d1a184aae" path="/var/lib/kubelet/pods/d6e28c13-f0e4-4ebe-a8f7-192d1a184aae/volumes" Nov 02 20:37:02.395665 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:02.395635 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" Nov 02 20:37:02.396123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:02.395629 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl" event={"ID":"3e3c0362-fdcd-4801-9634-657f6ad5dcae","Type":"ContainerDied","Data":"77baef90a5d3065767d25868dc1fac89b25898f1f1e754b471c07a147432f7bd"} Nov 02 20:37:02.396123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:02.395790 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77baef90a5d3065767d25868dc1fac89b25898f1f1e754b471c07a147432f7bd" Nov 02 20:37:03.152017 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.151985 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d"] Nov 02 20:37:03.152711 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.152696 4971 cpu_manager.go:414] "containerMap: removing container" podUID="3e3c0362-fdcd-4801-9634-657f6ad5dcae" containerName="revoke-keys" Nov 02 20:37:03.152766 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.152713 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e3c0362-fdcd-4801-9634-657f6ad5dcae" containerName="revoke-keys" Nov 02 20:37:03.152766 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.152727 4971 cpu_manager.go:414] "containerMap: removing container" podUID="39e0a7df-c53e-4fbd-8b6e-90d2adb023c1" containerName="revoke-keys" Nov 02 20:37:03.152766 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.152732 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e0a7df-c53e-4fbd-8b6e-90d2adb023c1" containerName="revoke-keys" Nov 02 20:37:03.152894 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.152884 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e3c0362-fdcd-4801-9634-657f6ad5dcae" containerName="revoke-keys" Nov 02 20:37:03.152928 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.152896 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e0a7df-c53e-4fbd-8b6e-90d2adb023c1" containerName="revoke-keys" Nov 02 20:37:03.156747 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.156728 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" Nov 02 20:37:03.184613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.184584 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqg6p\" (UniqueName: \"kubernetes.io/projected/28f72fa1-0c92-475d-9231-0c4aeac59f50-kube-api-access-jqg6p\") pod \"maas-api-revoke-sub-000c7454-66r7d\" (UID: \"28f72fa1-0c92-475d-9231-0c4aeac59f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" Nov 02 20:37:03.184743 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.184633 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f72fa1-0c92-475d-9231-0c4aeac59f50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-000c7454-66r7d\" (UID: \"28f72fa1-0c92-475d-9231-0c4aeac59f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" Nov 02 20:37:03.184864 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.184850 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f72fa1-0c92-475d-9231-0c4aeac59f50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-000c7454-66r7d\" (UID: \"28f72fa1-0c92-475d-9231-0c4aeac59f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" Nov 02 20:37:03.211970 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.211941 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d"] Nov 02 20:37:03.251239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.251211 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:37:03.251448 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.251272 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:37:03.286395 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.286374 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f72fa1-0c92-475d-9231-0c4aeac59f50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-000c7454-66r7d\" (UID: \"28f72fa1-0c92-475d-9231-0c4aeac59f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" Nov 02 20:37:03.286504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.286492 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqg6p\" (UniqueName: \"kubernetes.io/projected/28f72fa1-0c92-475d-9231-0c4aeac59f50-kube-api-access-jqg6p\") pod \"maas-api-revoke-sub-000c7454-66r7d\" (UID: \"28f72fa1-0c92-475d-9231-0c4aeac59f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" Nov 02 20:37:03.286557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.286515 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f72fa1-0c92-475d-9231-0c4aeac59f50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-000c7454-66r7d\" (UID: \"28f72fa1-0c92-475d-9231-0c4aeac59f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" Nov 02 20:37:03.287136 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.287115 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f72fa1-0c92-475d-9231-0c4aeac59f50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-000c7454-66r7d\" (UID: \"28f72fa1-0c92-475d-9231-0c4aeac59f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" Nov 02 20:37:03.288899 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.288876 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f72fa1-0c92-475d-9231-0c4aeac59f50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-000c7454-66r7d\" (UID: \"28f72fa1-0c92-475d-9231-0c4aeac59f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" Nov 02 20:37:03.340552 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:03.340520 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:37:03.344669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.344646 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqg6p\" (UniqueName: \"kubernetes.io/projected/28f72fa1-0c92-475d-9231-0c4aeac59f50-kube-api-access-jqg6p\") pod \"maas-api-revoke-sub-000c7454-66r7d\" (UID: \"28f72fa1-0c92-475d-9231-0c4aeac59f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" Nov 02 20:37:03.472440 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.472378 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" Nov 02 20:37:03.859601 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:03.859566 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f72fa1_0c92_475d_9231_0c4aeac59f50.slice/crio-0a5f18a01189914b3ff919618bf90ac661ee11266428bd97b038ef40fc628809 WatchSource:0}: Error finding container 0a5f18a01189914b3ff919618bf90ac661ee11266428bd97b038ef40fc628809: Status 404 returned error can't find the container with id 0a5f18a01189914b3ff919618bf90ac661ee11266428bd97b038ef40fc628809 Nov 02 20:37:03.872643 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:03.872620 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d"] Nov 02 20:37:04.228389 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:04.228340 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:04.415444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:04.415363 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" event={"ID":"28f72fa1-0c92-475d-9231-0c4aeac59f50","Type":"ContainerStarted","Data":"59a13af0e62d783234ed9ffc6e963a4659bfd4c851e6f8a574fc11795bd9db43"} Nov 02 20:37:04.415444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:04.415417 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" event={"ID":"28f72fa1-0c92-475d-9231-0c4aeac59f50","Type":"ContainerStarted","Data":"0a5f18a01189914b3ff919618bf90ac661ee11266428bd97b038ef40fc628809"} Nov 02 20:37:04.528722 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:04.528668 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" podStartSLOduration=1.52865354 podStartE2EDuration="1.52865354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:04.524503979 +0000 UTC m=+1352.814160530" watchObservedRunningTime="2026-11-02 20:37:04.52865354 +0000 UTC m=+1352.818310067" Nov 02 20:37:05.426970 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:05.426939 4971 generic.go:356] "Generic (PLEG): container finished" podID="28f72fa1-0c92-475d-9231-0c4aeac59f50" containerID="59a13af0e62d783234ed9ffc6e963a4659bfd4c851e6f8a574fc11795bd9db43" exitCode=0 Nov 02 20:37:05.427239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:05.427025 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" event={"ID":"28f72fa1-0c92-475d-9231-0c4aeac59f50","Type":"ContainerDied","Data":"59a13af0e62d783234ed9ffc6e963a4659bfd4c851e6f8a574fc11795bd9db43"} Nov 02 20:37:06.049949 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:06.049912 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb"] Nov 02 20:37:06.095993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:06.095965 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c9cffb8-4b5jb"] Nov 02 20:37:06.611694 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:06.611667 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" Nov 02 20:37:06.736398 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:06.736319 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28f72fa1-0c92-475d-9231-0c4aeac59f50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f72fa1-0c92-475d-9231-0c4aeac59f50-maas-api-xks-ca\") pod \"28f72fa1-0c92-475d-9231-0c4aeac59f50\" (UID: \"28f72fa1-0c92-475d-9231-0c4aeac59f50\") " Nov 02 20:37:06.736544 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:06.736431 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f72fa1-0c92-475d-9231-0c4aeac59f50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f72fa1-0c92-475d-9231-0c4aeac59f50-maas-api-openshift-ca\") pod \"28f72fa1-0c92-475d-9231-0c4aeac59f50\" (UID: \"28f72fa1-0c92-475d-9231-0c4aeac59f50\") " Nov 02 20:37:06.736544 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:06.736482 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f72fa1-0c92-475d-9231-0c4aeac59f50-kube-api-access-jqg6p\" (UniqueName: \"kubernetes.io/projected/28f72fa1-0c92-475d-9231-0c4aeac59f50-kube-api-access-jqg6p\") pod \"28f72fa1-0c92-475d-9231-0c4aeac59f50\" (UID: \"28f72fa1-0c92-475d-9231-0c4aeac59f50\") " Nov 02 20:37:06.736863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:06.736831 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f72fa1-0c92-475d-9231-0c4aeac59f50-maas-api-openshift-ca" pod "28f72fa1-0c92-475d-9231-0c4aeac59f50" (UID: "28f72fa1-0c92-475d-9231-0c4aeac59f50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:06.737399 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:06.737374 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f72fa1-0c92-475d-9231-0c4aeac59f50-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:06.738975 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:06.738946 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f72fa1-0c92-475d-9231-0c4aeac59f50-kube-api-access-jqg6p" pod "28f72fa1-0c92-475d-9231-0c4aeac59f50" (UID: "28f72fa1-0c92-475d-9231-0c4aeac59f50"). InnerVolumeSpecName "kube-api-access-jqg6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:06.742180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:06.742158 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f72fa1-0c92-475d-9231-0c4aeac59f50-maas-api-xks-ca" pod "28f72fa1-0c92-475d-9231-0c4aeac59f50" (UID: "28f72fa1-0c92-475d-9231-0c4aeac59f50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:06.838160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:06.838135 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f72fa1-0c92-475d-9231-0c4aeac59f50-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:06.838160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:06.838159 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqg6p\" (UniqueName: \"kubernetes.io/projected/28f72fa1-0c92-475d-9231-0c4aeac59f50-kube-api-access-jqg6p\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:07.048974 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.048902 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2"] Nov 02 20:37:07.053621 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.053596 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-446ad717-l2xc2"] Nov 02 20:37:07.099027 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.098999 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc"] Nov 02 20:37:07.104715 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.104689 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f4518cf-qv7lc"] Nov 02 20:37:07.446320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.446284 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" event={"ID":"28f72fa1-0c92-475d-9231-0c4aeac59f50","Type":"ContainerDied","Data":"0a5f18a01189914b3ff919618bf90ac661ee11266428bd97b038ef40fc628809"} Nov 02 20:37:07.446320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.446307 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d" Nov 02 20:37:07.446526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.446336 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a5f18a01189914b3ff919618bf90ac661ee11266428bd97b038ef40fc628809" Nov 02 20:37:07.867826 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.867792 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4459c350-cd02-4b0e-b28f-98e0ead6a077" path="/var/lib/kubelet/pods/4459c350-cd02-4b0e-b28f-98e0ead6a077/volumes" Nov 02 20:37:07.868339 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.868314 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0ae96b-7041-4e4b-bc18-84d76a5d9e51" path="/var/lib/kubelet/pods/da0ae96b-7041-4e4b-bc18-84d76a5d9e51/volumes" Nov 02 20:37:07.868783 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.868763 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f72d94b0-291c-421a-90c1-30b48b5ac842" path="/var/lib/kubelet/pods/f72d94b0-291c-421a-90c1-30b48b5ac842/volumes" Nov 02 20:37:07.883664 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.883638 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb"] Nov 02 20:37:07.884305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.884291 4971 cpu_manager.go:414] "containerMap: removing container" podUID="28f72fa1-0c92-475d-9231-0c4aeac59f50" containerName="revoke-keys" Nov 02 20:37:07.884360 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.884306 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f72fa1-0c92-475d-9231-0c4aeac59f50" containerName="revoke-keys" Nov 02 20:37:07.884442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.884433 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f72fa1-0c92-475d-9231-0c4aeac59f50" containerName="revoke-keys" Nov 02 20:37:07.889372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.889356 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" Nov 02 20:37:07.960372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.943473 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb"] Nov 02 20:37:07.960372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.944449 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:37:07.960372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.946903 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:37:07.960372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.951227 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8abc64d-820c-4e63-b288-e174393765dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a45639c6-lbwpb\" (UID: \"b8abc64d-820c-4e63-b288-e174393765dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" Nov 02 20:37:07.960372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.951349 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxqcv\" (UniqueName: \"kubernetes.io/projected/b8abc64d-820c-4e63-b288-e174393765dc-kube-api-access-xxqcv\") pod \"maas-api-revoke-sub-a45639c6-lbwpb\" (UID: \"b8abc64d-820c-4e63-b288-e174393765dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" Nov 02 20:37:07.960372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:07.951480 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8abc64d-820c-4e63-b288-e174393765dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a45639c6-lbwpb\" (UID: \"b8abc64d-820c-4e63-b288-e174393765dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" Nov 02 20:37:08.053133 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:08.053063 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8abc64d-820c-4e63-b288-e174393765dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a45639c6-lbwpb\" (UID: \"b8abc64d-820c-4e63-b288-e174393765dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" Nov 02 20:37:08.053302 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:08.053206 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxqcv\" (UniqueName: \"kubernetes.io/projected/b8abc64d-820c-4e63-b288-e174393765dc-kube-api-access-xxqcv\") pod \"maas-api-revoke-sub-a45639c6-lbwpb\" (UID: \"b8abc64d-820c-4e63-b288-e174393765dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" Nov 02 20:37:08.053372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:08.053331 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8abc64d-820c-4e63-b288-e174393765dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a45639c6-lbwpb\" (UID: \"b8abc64d-820c-4e63-b288-e174393765dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" Nov 02 20:37:08.053968 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:08.053949 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8abc64d-820c-4e63-b288-e174393765dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a45639c6-lbwpb\" (UID: \"b8abc64d-820c-4e63-b288-e174393765dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" Nov 02 20:37:08.055719 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:08.055697 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8abc64d-820c-4e63-b288-e174393765dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a45639c6-lbwpb\" (UID: \"b8abc64d-820c-4e63-b288-e174393765dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" Nov 02 20:37:08.111074 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:08.111048 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxqcv\" (UniqueName: \"kubernetes.io/projected/b8abc64d-820c-4e63-b288-e174393765dc-kube-api-access-xxqcv\") pod \"maas-api-revoke-sub-a45639c6-lbwpb\" (UID: \"b8abc64d-820c-4e63-b288-e174393765dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" Nov 02 20:37:08.203136 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:08.203045 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" Nov 02 20:37:08.362902 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:08.362869 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr"] Nov 02 20:37:08.363170 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:08.363142 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" podUID="e271f874-deb0-487e-a4bc-ee0f520bb501" containerName="istio-proxy" containerID="cri-o://03b86e5c7290400c2b5186ef67d2287dda37bc042fe5423787f9086982991b46" gracePeriod=30 Nov 02 20:37:08.368155 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:08.368125 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp"] Nov 02 20:37:08.368395 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:08.368354 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" podUID="b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" containerName="istio-proxy" containerID="cri-o://cf5042cffb0b6ec65d73a4cb8b8697be704113ffebdd1b558d161b99ad6d460c" gracePeriod=30 Nov 02 20:37:08.373252 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:08.373229 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt"] Nov 02 20:37:08.373398 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:08.373381 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" podUID="817f7e35-0db0-493e-a3af-825e1818eb18" containerName="istio-proxy" containerID="cri-o://3052b3b7f6606fb2fd26bc1c17f042031c0fbcd9aa74be8ef86a3987798375eb" gracePeriod=30 Nov 02 20:37:08.408503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:08.408476 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv"] Nov 02 20:37:08.408667 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:08.408636 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" podUID="a02d7838-6668-43b5-b0d8-4e5af366edf1" containerName="istio-proxy" containerID="cri-o://aba37b744deaecfe6f274adca6819cd64d018045121794e71f96078a8bfd05b5" gracePeriod=30 Nov 02 20:37:08.585406 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:08.585372 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8abc64d_820c_4e63_b288_e174393765dc.slice/crio-6e7d576dce41fc1e2f74f4bd8f79907583c08f87e94a2d88622924b8d24823d7 WatchSource:0}: Error finding container 6e7d576dce41fc1e2f74f4bd8f79907583c08f87e94a2d88622924b8d24823d7: Status 404 returned error can't find the container with id 6e7d576dce41fc1e2f74f4bd8f79907583c08f87e94a2d88622924b8d24823d7 Nov 02 20:37:08.586589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:08.586561 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb"] Nov 02 20:37:09.042740 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.042702 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:09.051339 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.051297 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm"] Nov 02 20:37:09.056275 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.056254 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa722c67-nkqzm"] Nov 02 20:37:09.266033 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.265997 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv"] Nov 02 20:37:09.270532 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.270512 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" Nov 02 20:37:09.324498 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.324470 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv"] Nov 02 20:37:09.373893 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.373868 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl426\" (UniqueName: \"kubernetes.io/projected/ef5c9309-85f8-4d06-9ede-25224452834d-kube-api-access-nl426\") pod \"maas-api-revoke-sub-6e232924-4glqv\" (UID: \"ef5c9309-85f8-4d06-9ede-25224452834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" Nov 02 20:37:09.374069 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.374051 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef5c9309-85f8-4d06-9ede-25224452834d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e232924-4glqv\" (UID: \"ef5c9309-85f8-4d06-9ede-25224452834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" Nov 02 20:37:09.374227 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.374199 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef5c9309-85f8-4d06-9ede-25224452834d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e232924-4glqv\" (UID: \"ef5c9309-85f8-4d06-9ede-25224452834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" Nov 02 20:37:09.467885 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.467857 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" event={"ID":"b8abc64d-820c-4e63-b288-e174393765dc","Type":"ContainerStarted","Data":"a62329ac90b782238c5d7b240ae8b1e143a51e1801ce393f21cdcab15d0be172"} Nov 02 20:37:09.468067 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.467904 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" event={"ID":"b8abc64d-820c-4e63-b288-e174393765dc","Type":"ContainerStarted","Data":"6e7d576dce41fc1e2f74f4bd8f79907583c08f87e94a2d88622924b8d24823d7"} Nov 02 20:37:09.475329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.475304 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef5c9309-85f8-4d06-9ede-25224452834d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e232924-4glqv\" (UID: \"ef5c9309-85f8-4d06-9ede-25224452834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" Nov 02 20:37:09.475433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.475352 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef5c9309-85f8-4d06-9ede-25224452834d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e232924-4glqv\" (UID: \"ef5c9309-85f8-4d06-9ede-25224452834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" Nov 02 20:37:09.475433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.475421 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl426\" (UniqueName: \"kubernetes.io/projected/ef5c9309-85f8-4d06-9ede-25224452834d-kube-api-access-nl426\") pod \"maas-api-revoke-sub-6e232924-4glqv\" (UID: \"ef5c9309-85f8-4d06-9ede-25224452834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" Nov 02 20:37:09.475983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.475960 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef5c9309-85f8-4d06-9ede-25224452834d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e232924-4glqv\" (UID: \"ef5c9309-85f8-4d06-9ede-25224452834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" Nov 02 20:37:09.477816 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.477797 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef5c9309-85f8-4d06-9ede-25224452834d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e232924-4glqv\" (UID: \"ef5c9309-85f8-4d06-9ede-25224452834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" Nov 02 20:37:09.533847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.533791 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl426\" (UniqueName: \"kubernetes.io/projected/ef5c9309-85f8-4d06-9ede-25224452834d-kube-api-access-nl426\") pod \"maas-api-revoke-sub-6e232924-4glqv\" (UID: \"ef5c9309-85f8-4d06-9ede-25224452834d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" Nov 02 20:37:09.581353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.581305 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" podStartSLOduration=2.581292323 podStartE2EDuration="2.581292323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:09.576373281 +0000 UTC m=+1357.866029813" watchObservedRunningTime="2026-11-02 20:37:09.581292323 +0000 UTC m=+1357.870948850" Nov 02 20:37:09.585492 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.585471 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" Nov 02 20:37:09.869525 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.869497 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39568bf0-9145-4591-856a-80feed1768aa" path="/var/lib/kubelet/pods/39568bf0-9145-4591-856a-80feed1768aa/volumes" Nov 02 20:37:09.962830 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:09.962802 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5c9309_85f8_4d06_9ede_25224452834d.slice/crio-190fe1f9d80f06a0c7f1fe6c9ffbb72ae4c5a2ae4eaf9550ebe880d5062aea2b WatchSource:0}: Error finding container 190fe1f9d80f06a0c7f1fe6c9ffbb72ae4c5a2ae4eaf9550ebe880d5062aea2b: Status 404 returned error can't find the container with id 190fe1f9d80f06a0c7f1fe6c9ffbb72ae4c5a2ae4eaf9550ebe880d5062aea2b Nov 02 20:37:09.964753 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:09.964692 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv"] Nov 02 20:37:10.478030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:10.477961 4971 generic.go:356] "Generic (PLEG): container finished" podID="b8abc64d-820c-4e63-b288-e174393765dc" containerID="a62329ac90b782238c5d7b240ae8b1e143a51e1801ce393f21cdcab15d0be172" exitCode=0 Nov 02 20:37:10.478412 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:10.478050 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" event={"ID":"b8abc64d-820c-4e63-b288-e174393765dc","Type":"ContainerDied","Data":"a62329ac90b782238c5d7b240ae8b1e143a51e1801ce393f21cdcab15d0be172"} Nov 02 20:37:10.479771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:10.479751 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" event={"ID":"ef5c9309-85f8-4d06-9ede-25224452834d","Type":"ContainerStarted","Data":"4b0364b5d023fb3d5c6450537b8fc969e343eee7fdc3b988ee65aa4c836f3c06"} Nov 02 20:37:10.479865 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:10.479791 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" event={"ID":"ef5c9309-85f8-4d06-9ede-25224452834d","Type":"ContainerStarted","Data":"190fe1f9d80f06a0c7f1fe6c9ffbb72ae4c5a2ae4eaf9550ebe880d5062aea2b"} Nov 02 20:37:10.707759 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:10.707708 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" podStartSLOduration=1.707694584 podStartE2EDuration="1.707694584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:10.702995502 +0000 UTC m=+1358.992652029" watchObservedRunningTime="2026-11-02 20:37:10.707694584 +0000 UTC m=+1358.997351112" Nov 02 20:37:11.491875 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:11.491840 4971 generic.go:356] "Generic (PLEG): container finished" podID="ef5c9309-85f8-4d06-9ede-25224452834d" containerID="4b0364b5d023fb3d5c6450537b8fc969e343eee7fdc3b988ee65aa4c836f3c06" exitCode=0 Nov 02 20:37:11.492310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:11.491876 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" event={"ID":"ef5c9309-85f8-4d06-9ede-25224452834d","Type":"ContainerDied","Data":"4b0364b5d023fb3d5c6450537b8fc969e343eee7fdc3b988ee65aa4c836f3c06"} Nov 02 20:37:11.665118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:11.665067 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" Nov 02 20:37:11.809211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:11.808670 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8abc64d-820c-4e63-b288-e174393765dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8abc64d-820c-4e63-b288-e174393765dc-maas-api-openshift-ca\") pod \"b8abc64d-820c-4e63-b288-e174393765dc\" (UID: \"b8abc64d-820c-4e63-b288-e174393765dc\") " Nov 02 20:37:11.809211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:11.809054 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8abc64d-820c-4e63-b288-e174393765dc-kube-api-access-xxqcv\" (UniqueName: \"kubernetes.io/projected/b8abc64d-820c-4e63-b288-e174393765dc-kube-api-access-xxqcv\") pod \"b8abc64d-820c-4e63-b288-e174393765dc\" (UID: \"b8abc64d-820c-4e63-b288-e174393765dc\") " Nov 02 20:37:11.810073 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:11.810041 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8abc64d-820c-4e63-b288-e174393765dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8abc64d-820c-4e63-b288-e174393765dc-maas-api-xks-ca\") pod \"b8abc64d-820c-4e63-b288-e174393765dc\" (UID: \"b8abc64d-820c-4e63-b288-e174393765dc\") " Nov 02 20:37:11.810284 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:11.810181 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8abc64d-820c-4e63-b288-e174393765dc-maas-api-openshift-ca" pod "b8abc64d-820c-4e63-b288-e174393765dc" (UID: "b8abc64d-820c-4e63-b288-e174393765dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:11.810954 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:11.810908 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8abc64d-820c-4e63-b288-e174393765dc-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:11.814404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:11.814375 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8abc64d-820c-4e63-b288-e174393765dc-kube-api-access-xxqcv" pod "b8abc64d-820c-4e63-b288-e174393765dc" (UID: "b8abc64d-820c-4e63-b288-e174393765dc"). InnerVolumeSpecName "kube-api-access-xxqcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:11.814503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:11.814467 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8abc64d-820c-4e63-b288-e174393765dc-maas-api-xks-ca" pod "b8abc64d-820c-4e63-b288-e174393765dc" (UID: "b8abc64d-820c-4e63-b288-e174393765dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:11.912184 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:11.912148 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxqcv\" (UniqueName: \"kubernetes.io/projected/b8abc64d-820c-4e63-b288-e174393765dc-kube-api-access-xxqcv\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:11.912184 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:11.912181 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8abc64d-820c-4e63-b288-e174393765dc-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:12.502152 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.502121 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" Nov 02 20:37:12.502571 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.502156 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb" event={"ID":"b8abc64d-820c-4e63-b288-e174393765dc","Type":"ContainerDied","Data":"6e7d576dce41fc1e2f74f4bd8f79907583c08f87e94a2d88622924b8d24823d7"} Nov 02 20:37:12.502571 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.502220 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e7d576dce41fc1e2f74f4bd8f79907583c08f87e94a2d88622924b8d24823d7" Nov 02 20:37:12.609603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.609569 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4"] Nov 02 20:37:12.610681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.610654 4971 cpu_manager.go:414] "containerMap: removing container" podUID="b8abc64d-820c-4e63-b288-e174393765dc" containerName="revoke-keys" Nov 02 20:37:12.610870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.610852 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8abc64d-820c-4e63-b288-e174393765dc" containerName="revoke-keys" Nov 02 20:37:12.611320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.611299 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8abc64d-820c-4e63-b288-e174393765dc" containerName="revoke-keys" Nov 02 20:37:12.615784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.615764 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4" Nov 02 20:37:12.644485 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.644462 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4"] Nov 02 20:37:12.701042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.701007 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" Nov 02 20:37:12.724313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.724287 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d887a152-fb99-48f3-b702-779ce22c4e59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16fbf498-6hhd4\" (UID: \"d887a152-fb99-48f3-b702-779ce22c4e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4" Nov 02 20:37:12.724551 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.724525 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kn78\" (UniqueName: \"kubernetes.io/projected/d887a152-fb99-48f3-b702-779ce22c4e59-kube-api-access-4kn78\") pod \"maas-api-revoke-sub-16fbf498-6hhd4\" (UID: \"d887a152-fb99-48f3-b702-779ce22c4e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4" Nov 02 20:37:12.724787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.724771 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d887a152-fb99-48f3-b702-779ce22c4e59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16fbf498-6hhd4\" (UID: \"d887a152-fb99-48f3-b702-779ce22c4e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4" Nov 02 20:37:12.825430 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.825365 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef5c9309-85f8-4d06-9ede-25224452834d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef5c9309-85f8-4d06-9ede-25224452834d-maas-api-xks-ca\") pod \"ef5c9309-85f8-4d06-9ede-25224452834d\" (UID: \"ef5c9309-85f8-4d06-9ede-25224452834d\") " Nov 02 20:37:12.825430 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.825415 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef5c9309-85f8-4d06-9ede-25224452834d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef5c9309-85f8-4d06-9ede-25224452834d-maas-api-openshift-ca\") pod \"ef5c9309-85f8-4d06-9ede-25224452834d\" (UID: \"ef5c9309-85f8-4d06-9ede-25224452834d\") " Nov 02 20:37:12.825606 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.825447 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5c9309-85f8-4d06-9ede-25224452834d-kube-api-access-nl426\" (UniqueName: \"kubernetes.io/projected/ef5c9309-85f8-4d06-9ede-25224452834d-kube-api-access-nl426\") pod \"ef5c9309-85f8-4d06-9ede-25224452834d\" (UID: \"ef5c9309-85f8-4d06-9ede-25224452834d\") " Nov 02 20:37:12.825796 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.825771 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d887a152-fb99-48f3-b702-779ce22c4e59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16fbf498-6hhd4\" (UID: \"d887a152-fb99-48f3-b702-779ce22c4e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4" Nov 02 20:37:12.825880 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.825821 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5c9309-85f8-4d06-9ede-25224452834d-maas-api-openshift-ca" pod "ef5c9309-85f8-4d06-9ede-25224452834d" (UID: "ef5c9309-85f8-4d06-9ede-25224452834d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:12.825985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.825968 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d887a152-fb99-48f3-b702-779ce22c4e59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16fbf498-6hhd4\" (UID: \"d887a152-fb99-48f3-b702-779ce22c4e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4" Nov 02 20:37:12.826052 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.825997 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kn78\" (UniqueName: \"kubernetes.io/projected/d887a152-fb99-48f3-b702-779ce22c4e59-kube-api-access-4kn78\") pod \"maas-api-revoke-sub-16fbf498-6hhd4\" (UID: \"d887a152-fb99-48f3-b702-779ce22c4e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4" Nov 02 20:37:12.826373 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.826352 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef5c9309-85f8-4d06-9ede-25224452834d-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:12.826876 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.826851 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d887a152-fb99-48f3-b702-779ce22c4e59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16fbf498-6hhd4\" (UID: \"d887a152-fb99-48f3-b702-779ce22c4e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4" Nov 02 20:37:12.827900 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.827872 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5c9309-85f8-4d06-9ede-25224452834d-kube-api-access-nl426" pod "ef5c9309-85f8-4d06-9ede-25224452834d" (UID: "ef5c9309-85f8-4d06-9ede-25224452834d"). InnerVolumeSpecName "kube-api-access-nl426". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:12.828257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.828235 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef5c9309-85f8-4d06-9ede-25224452834d-maas-api-xks-ca" pod "ef5c9309-85f8-4d06-9ede-25224452834d" (UID: "ef5c9309-85f8-4d06-9ede-25224452834d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:12.828558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.828541 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d887a152-fb99-48f3-b702-779ce22c4e59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16fbf498-6hhd4\" (UID: \"d887a152-fb99-48f3-b702-779ce22c4e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4" Nov 02 20:37:12.885140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.884688 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kn78\" (UniqueName: \"kubernetes.io/projected/d887a152-fb99-48f3-b702-779ce22c4e59-kube-api-access-4kn78\") pod \"maas-api-revoke-sub-16fbf498-6hhd4\" (UID: \"d887a152-fb99-48f3-b702-779ce22c4e59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4" Nov 02 20:37:12.928337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.928310 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef5c9309-85f8-4d06-9ede-25224452834d-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:12.928337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.928336 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl426\" (UniqueName: \"kubernetes.io/projected/ef5c9309-85f8-4d06-9ede-25224452834d-kube-api-access-nl426\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:12.937192 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:12.937170 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4" Nov 02 20:37:13.278058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.278025 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx"] Nov 02 20:37:13.278987 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.278963 4971 cpu_manager.go:414] "containerMap: removing container" podUID="ef5c9309-85f8-4d06-9ede-25224452834d" containerName="revoke-keys" Nov 02 20:37:13.278987 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.278983 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5c9309-85f8-4d06-9ede-25224452834d" containerName="revoke-keys" Nov 02 20:37:13.279192 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.279147 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5c9309-85f8-4d06-9ede-25224452834d" containerName="revoke-keys" Nov 02 20:37:13.283178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.283153 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx" Nov 02 20:37:13.329624 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:13.327431 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd887a152_fb99_48f3_b702_779ce22c4e59.slice/crio-9cfdbdfdd8aca7d49da537f7023c10b1571144f84b9b33cc93b4e8d52b98e944 WatchSource:0}: Error finding container 9cfdbdfdd8aca7d49da537f7023c10b1571144f84b9b33cc93b4e8d52b98e944: Status 404 returned error can't find the container with id 9cfdbdfdd8aca7d49da537f7023c10b1571144f84b9b33cc93b4e8d52b98e944 Nov 02 20:37:13.333401 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.333372 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4"] Nov 02 20:37:13.341043 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.340828 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx"] Nov 02 20:37:13.436321 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.436293 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16247d79-tvpfx\" (UID: \"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx" Nov 02 20:37:13.436509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.436362 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16247d79-tvpfx\" (UID: \"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx" Nov 02 20:37:13.436628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.436610 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xthx6\" (UniqueName: \"kubernetes.io/projected/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-kube-api-access-xthx6\") pod \"maas-api-revoke-sub-16247d79-tvpfx\" (UID: \"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx" Nov 02 20:37:13.512603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.512580 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" Nov 02 20:37:13.512956 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.512579 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv" event={"ID":"ef5c9309-85f8-4d06-9ede-25224452834d","Type":"ContainerDied","Data":"190fe1f9d80f06a0c7f1fe6c9ffbb72ae4c5a2ae4eaf9550ebe880d5062aea2b"} Nov 02 20:37:13.512956 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.512714 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="190fe1f9d80f06a0c7f1fe6c9ffbb72ae4c5a2ae4eaf9550ebe880d5062aea2b" Nov 02 20:37:13.519733 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.519705 4971 generic.go:356] "Generic (PLEG): container finished" podID="b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" containerID="cf5042cffb0b6ec65d73a4cb8b8697be704113ffebdd1b558d161b99ad6d460c" exitCode=0 Nov 02 20:37:13.519842 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.519780 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" event={"ID":"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe","Type":"ContainerDied","Data":"cf5042cffb0b6ec65d73a4cb8b8697be704113ffebdd1b558d161b99ad6d460c"} Nov 02 20:37:13.521593 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.521571 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4" event={"ID":"d887a152-fb99-48f3-b702-779ce22c4e59","Type":"ContainerStarted","Data":"9cfdbdfdd8aca7d49da537f7023c10b1571144f84b9b33cc93b4e8d52b98e944"} Nov 02 20:37:13.523683 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.523647 4971 generic.go:356] "Generic (PLEG): container finished" podID="817f7e35-0db0-493e-a3af-825e1818eb18" containerID="3052b3b7f6606fb2fd26bc1c17f042031c0fbcd9aa74be8ef86a3987798375eb" exitCode=0 Nov 02 20:37:13.523792 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.523726 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" event={"ID":"817f7e35-0db0-493e-a3af-825e1818eb18","Type":"ContainerDied","Data":"3052b3b7f6606fb2fd26bc1c17f042031c0fbcd9aa74be8ef86a3987798375eb"} Nov 02 20:37:13.525753 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.525734 4971 generic.go:356] "Generic (PLEG): container finished" podID="e271f874-deb0-487e-a4bc-ee0f520bb501" containerID="03b86e5c7290400c2b5186ef67d2287dda37bc042fe5423787f9086982991b46" exitCode=0 Nov 02 20:37:13.525872 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.525802 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" event={"ID":"e271f874-deb0-487e-a4bc-ee0f520bb501","Type":"ContainerDied","Data":"03b86e5c7290400c2b5186ef67d2287dda37bc042fe5423787f9086982991b46"} Nov 02 20:37:13.540440 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.538816 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16247d79-tvpfx\" (UID: \"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx" Nov 02 20:37:13.540440 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.539059 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xthx6\" (UniqueName: \"kubernetes.io/projected/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-kube-api-access-xthx6\") pod \"maas-api-revoke-sub-16247d79-tvpfx\" (UID: \"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx" Nov 02 20:37:13.540440 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.539264 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16247d79-tvpfx\" (UID: \"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx" Nov 02 20:37:13.541209 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.541112 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16247d79-tvpfx\" (UID: \"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx" Nov 02 20:37:13.548132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.546517 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16247d79-tvpfx\" (UID: \"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx" Nov 02 20:37:13.606451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.606417 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xthx6\" (UniqueName: \"kubernetes.io/projected/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-kube-api-access-xthx6\") pod \"maas-api-revoke-sub-16247d79-tvpfx\" (UID: \"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx" Nov 02 20:37:13.757771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.757743 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:37:13.761676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.761652 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:37:13.796593 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.796535 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:37:13.809048 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.809029 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:37:13.898337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.898302 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx" Nov 02 20:37:13.916289 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.916074 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:13.948357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948323 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-workload-certs\") pod \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " Nov 02 20:37:13.948504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948390 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-credential-socket\") pod \"817f7e35-0db0-493e-a3af-825e1818eb18\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " Nov 02 20:37:13.948504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948433 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/817f7e35-0db0-493e-a3af-825e1818eb18-kube-api-access-jkm5w\" (UniqueName: \"kubernetes.io/projected/817f7e35-0db0-493e-a3af-825e1818eb18-kube-api-access-jkm5w\") pod \"817f7e35-0db0-493e-a3af-825e1818eb18\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " Nov 02 20:37:13.948645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948506 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/817f7e35-0db0-493e-a3af-825e1818eb18-istio-token\" (UniqueName: \"kubernetes.io/projected/817f7e35-0db0-493e-a3af-825e1818eb18-istio-token\") pod \"817f7e35-0db0-493e-a3af-825e1818eb18\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " Nov 02 20:37:13.948645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948536 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02d7838-6668-43b5-b0d8-4e5af366edf1-kube-api-access-cm8ng\" (UniqueName: \"kubernetes.io/projected/a02d7838-6668-43b5-b0d8-4e5af366edf1-kube-api-access-cm8ng\") pod \"a02d7838-6668-43b5-b0d8-4e5af366edf1\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " Nov 02 20:37:13.948645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948538 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-workload-certs" pod "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" (UID: "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.948645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948565 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istiod-ca-cert\") pod \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " Nov 02 20:37:13.948645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948610 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-podinfo\") pod \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " Nov 02 20:37:13.949198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948674 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/817f7e35-0db0-493e-a3af-825e1818eb18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/817f7e35-0db0-493e-a3af-825e1818eb18-istio-podinfo\") pod \"817f7e35-0db0-493e-a3af-825e1818eb18\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " Nov 02 20:37:13.949198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948710 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-podinfo\") pod \"e271f874-deb0-487e-a4bc-ee0f520bb501\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " Nov 02 20:37:13.949198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948738 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-workload-certs\") pod \"a02d7838-6668-43b5-b0d8-4e5af366edf1\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " Nov 02 20:37:13.949198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948789 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-token\" (UniqueName: \"kubernetes.io/projected/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-token\") pod \"a02d7838-6668-43b5-b0d8-4e5af366edf1\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " Nov 02 20:37:13.949198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948830 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-credential-socket\") pod \"a02d7838-6668-43b5-b0d8-4e5af366edf1\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " Nov 02 20:37:13.949198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948860 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-envoy\") pod \"a02d7838-6668-43b5-b0d8-4e5af366edf1\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " Nov 02 20:37:13.949198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948893 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-workload-socket\") pod \"a02d7838-6668-43b5-b0d8-4e5af366edf1\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " Nov 02 20:37:13.949198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948941 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-credential-socket\") pod \"e271f874-deb0-487e-a4bc-ee0f520bb501\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " Nov 02 20:37:13.949198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.948988 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a02d7838-6668-43b5-b0d8-4e5af366edf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02d7838-6668-43b5-b0d8-4e5af366edf1-istiod-ca-cert\") pod \"a02d7838-6668-43b5-b0d8-4e5af366edf1\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " Nov 02 20:37:13.949198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949020 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e271f874-deb0-487e-a4bc-ee0f520bb501-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e271f874-deb0-487e-a4bc-ee0f520bb501-istiod-ca-cert\") pod \"e271f874-deb0-487e-a4bc-ee0f520bb501\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " Nov 02 20:37:13.949198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949052 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-data\") pod \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " Nov 02 20:37:13.949198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949094 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-istio-envoy\") pod \"817f7e35-0db0-493e-a3af-825e1818eb18\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " Nov 02 20:37:13.949198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949129 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-envoy\") pod \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " Nov 02 20:37:13.949198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949193 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-data\") pod \"a02d7838-6668-43b5-b0d8-4e5af366edf1\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " Nov 02 20:37:13.949923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949209 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-credential-socket" pod "817f7e35-0db0-493e-a3af-825e1818eb18" (UID: "817f7e35-0db0-493e-a3af-825e1818eb18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.949923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949227 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-token\" (UniqueName: \"kubernetes.io/projected/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-token\") pod \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " Nov 02 20:37:13.949923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949257 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-envoy\") pod \"e271f874-deb0-487e-a4bc-ee0f520bb501\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " Nov 02 20:37:13.949923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949296 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-workload-socket\") pod \"817f7e35-0db0-493e-a3af-825e1818eb18\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " Nov 02 20:37:13.949923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949293 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-credential-socket" pod "a02d7838-6668-43b5-b0d8-4e5af366edf1" (UID: "a02d7838-6668-43b5-b0d8-4e5af366edf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.949923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949367 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-workload-certs\") pod \"817f7e35-0db0-493e-a3af-825e1818eb18\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " Nov 02 20:37:13.949923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949712 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-workload-certs\") pod \"e271f874-deb0-487e-a4bc-ee0f520bb501\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " Nov 02 20:37:13.949923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949755 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-kube-api-access-xsr9n\" (UniqueName: \"kubernetes.io/projected/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-kube-api-access-xsr9n\") pod \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " Nov 02 20:37:13.949923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949795 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-workload-socket\") pod \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " Nov 02 20:37:13.949923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949821 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-credential-socket\") pod \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\" (UID: \"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe\") " Nov 02 20:37:13.949923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949906 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-token\" (UniqueName: \"kubernetes.io/projected/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-token\") pod \"e271f874-deb0-487e-a4bc-ee0f520bb501\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " Nov 02 20:37:13.950824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949944 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/817f7e35-0db0-493e-a3af-825e1818eb18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/817f7e35-0db0-493e-a3af-825e1818eb18-istiod-ca-cert\") pod \"817f7e35-0db0-493e-a3af-825e1818eb18\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " Nov 02 20:37:13.950824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949979 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e271f874-deb0-487e-a4bc-ee0f520bb501-kube-api-access-hdcmq\" (UniqueName: \"kubernetes.io/projected/e271f874-deb0-487e-a4bc-ee0f520bb501-kube-api-access-hdcmq\") pod \"e271f874-deb0-487e-a4bc-ee0f520bb501\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " Nov 02 20:37:13.950824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949997 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-podinfo\") pod \"a02d7838-6668-43b5-b0d8-4e5af366edf1\" (UID: \"a02d7838-6668-43b5-b0d8-4e5af366edf1\") " Nov 02 20:37:13.950824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.950036 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-istio-data\") pod \"817f7e35-0db0-493e-a3af-825e1818eb18\" (UID: \"817f7e35-0db0-493e-a3af-825e1818eb18\") " Nov 02 20:37:13.950824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.950063 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-workload-socket\") pod \"e271f874-deb0-487e-a4bc-ee0f520bb501\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " Nov 02 20:37:13.950824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.950115 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-data\") pod \"e271f874-deb0-487e-a4bc-ee0f520bb501\" (UID: \"e271f874-deb0-487e-a4bc-ee0f520bb501\") " Nov 02 20:37:13.950824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.950722 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:13.950824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.950735 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:13.950824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.950745 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:13.953930 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949476 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-workload-certs" pod "a02d7838-6668-43b5-b0d8-4e5af366edf1" (UID: "a02d7838-6668-43b5-b0d8-4e5af366edf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.953930 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949511 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istiod-ca-cert" pod "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" (UID: "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:13.954122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.949519 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-workload-socket" pod "817f7e35-0db0-493e-a3af-825e1818eb18" (UID: "817f7e35-0db0-493e-a3af-825e1818eb18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.954122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.951131 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-data" pod "e271f874-deb0-487e-a4bc-ee0f520bb501" (UID: "e271f874-deb0-487e-a4bc-ee0f520bb501"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.954122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.953514 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-podinfo" pod "e271f874-deb0-487e-a4bc-ee0f520bb501" (UID: "e271f874-deb0-487e-a4bc-ee0f520bb501"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:37:13.954122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.953557 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-envoy" pod "a02d7838-6668-43b5-b0d8-4e5af366edf1" (UID: "a02d7838-6668-43b5-b0d8-4e5af366edf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.954122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.953761 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-workload-certs" pod "e271f874-deb0-487e-a4bc-ee0f520bb501" (UID: "e271f874-deb0-487e-a4bc-ee0f520bb501"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.954122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.953907 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-workload-certs" pod "817f7e35-0db0-493e-a3af-825e1818eb18" (UID: "817f7e35-0db0-493e-a3af-825e1818eb18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.954446 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.954182 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-credential-socket" pod "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" (UID: "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.954446 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.954276 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02d7838-6668-43b5-b0d8-4e5af366edf1-istiod-ca-cert" pod "a02d7838-6668-43b5-b0d8-4e5af366edf1" (UID: "a02d7838-6668-43b5-b0d8-4e5af366edf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:13.954554 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.954516 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-workload-socket" pod "a02d7838-6668-43b5-b0d8-4e5af366edf1" (UID: "a02d7838-6668-43b5-b0d8-4e5af366edf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.954655 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.954629 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-istio-envoy" pod "817f7e35-0db0-493e-a3af-825e1818eb18" (UID: "817f7e35-0db0-493e-a3af-825e1818eb18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.954710 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.954672 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-credential-socket" pod "e271f874-deb0-487e-a4bc-ee0f520bb501" (UID: "e271f874-deb0-487e-a4bc-ee0f520bb501"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.954860 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.954840 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-data" pod "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" (UID: "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.954951 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.954882 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e271f874-deb0-487e-a4bc-ee0f520bb501-istiod-ca-cert" pod "e271f874-deb0-487e-a4bc-ee0f520bb501" (UID: "e271f874-deb0-487e-a4bc-ee0f520bb501"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:13.955223 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.955199 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-workload-socket" pod "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" (UID: "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.955901 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.955413 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-istio-data" pod "817f7e35-0db0-493e-a3af-825e1818eb18" (UID: "817f7e35-0db0-493e-a3af-825e1818eb18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.955901 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.955619 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-workload-socket" pod "e271f874-deb0-487e-a4bc-ee0f520bb501" (UID: "e271f874-deb0-487e-a4bc-ee0f520bb501"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.955901 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.955800 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-data" pod "a02d7838-6668-43b5-b0d8-4e5af366edf1" (UID: "a02d7838-6668-43b5-b0d8-4e5af366edf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.955901 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.955878 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817f7e35-0db0-493e-a3af-825e1818eb18-istiod-ca-cert" pod "817f7e35-0db0-493e-a3af-825e1818eb18" (UID: "817f7e35-0db0-493e-a3af-825e1818eb18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:13.958763 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.957340 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-podinfo" pod "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" (UID: "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:37:13.959468 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.959222 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-envoy" pod "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" (UID: "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.959468 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.959255 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-envoy" pod "e271f874-deb0-487e-a4bc-ee0f520bb501" (UID: "e271f874-deb0-487e-a4bc-ee0f520bb501"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:13.959468 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.959417 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817f7e35-0db0-493e-a3af-825e1818eb18-istio-token" pod "817f7e35-0db0-493e-a3af-825e1818eb18" (UID: "817f7e35-0db0-493e-a3af-825e1818eb18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:13.959676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.959572 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e271f874-deb0-487e-a4bc-ee0f520bb501-kube-api-access-hdcmq" pod "e271f874-deb0-487e-a4bc-ee0f520bb501" (UID: "e271f874-deb0-487e-a4bc-ee0f520bb501"). InnerVolumeSpecName "kube-api-access-hdcmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:13.959676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.959605 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02d7838-6668-43b5-b0d8-4e5af366edf1-kube-api-access-cm8ng" pod "a02d7838-6668-43b5-b0d8-4e5af366edf1" (UID: "a02d7838-6668-43b5-b0d8-4e5af366edf1"). InnerVolumeSpecName "kube-api-access-cm8ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:13.959780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.959713 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-kube-api-access-xsr9n" pod "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" (UID: "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe"). InnerVolumeSpecName "kube-api-access-xsr9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:13.959833 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.959812 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-token" pod "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" (UID: "b0a8fe63-b184-4aa9-816b-eeb3c161ebbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:13.959904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.959886 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817f7e35-0db0-493e-a3af-825e1818eb18-kube-api-access-jkm5w" pod "817f7e35-0db0-493e-a3af-825e1818eb18" (UID: "817f7e35-0db0-493e-a3af-825e1818eb18"). InnerVolumeSpecName "kube-api-access-jkm5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:13.959978 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.959910 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-token" pod "a02d7838-6668-43b5-b0d8-4e5af366edf1" (UID: "a02d7838-6668-43b5-b0d8-4e5af366edf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:13.960032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.959975 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/817f7e35-0db0-493e-a3af-825e1818eb18-istio-podinfo" pod "817f7e35-0db0-493e-a3af-825e1818eb18" (UID: "817f7e35-0db0-493e-a3af-825e1818eb18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:37:13.961913 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.961775 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-podinfo" pod "a02d7838-6668-43b5-b0d8-4e5af366edf1" (UID: "a02d7838-6668-43b5-b0d8-4e5af366edf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:37:13.962392 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:13.962293 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-token" pod "e271f874-deb0-487e-a4bc-ee0f520bb501" (UID: "e271f874-deb0-487e-a4bc-ee0f520bb501"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:14.055063 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.054983 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e271f874-deb0-487e-a4bc-ee0f520bb501-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055063 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055024 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055063 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055040 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055063 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055053 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055066 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055098 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055114 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055128 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055144 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055156 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055170 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsr9n\" (UniqueName: \"kubernetes.io/projected/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-kube-api-access-xsr9n\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055184 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055196 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055208 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055222 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/817f7e35-0db0-493e-a3af-825e1818eb18-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055235 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdcmq\" (UniqueName: \"kubernetes.io/projected/e271f874-deb0-487e-a4bc-ee0f520bb501-kube-api-access-hdcmq\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055249 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055263 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/817f7e35-0db0-493e-a3af-825e1818eb18-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055277 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055290 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055304 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkm5w\" (UniqueName: \"kubernetes.io/projected/817f7e35-0db0-493e-a3af-825e1818eb18-kube-api-access-jkm5w\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055317 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/817f7e35-0db0-493e-a3af-825e1818eb18-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055330 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm8ng\" (UniqueName: \"kubernetes.io/projected/a02d7838-6668-43b5-b0d8-4e5af366edf1-kube-api-access-cm8ng\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055346 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055359 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055374 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/817f7e35-0db0-493e-a3af-825e1818eb18-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055388 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e271f874-deb0-487e-a4bc-ee0f520bb501-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055401 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.055400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055415 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.056393 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055428 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.056393 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055443 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02d7838-6668-43b5-b0d8-4e5af366edf1-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.056393 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055457 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e271f874-deb0-487e-a4bc-ee0f520bb501-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.056393 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.055470 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02d7838-6668-43b5-b0d8-4e5af366edf1-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:14.293824 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:14.293789 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240d9ecb_01ae_4e69_87c2_1ac5bdf6ba9a.slice/crio-8c7cf2f569196c7d08578fdaafff3b4ba8ed70c95546cbe564981f3766051730 WatchSource:0}: Error finding container 8c7cf2f569196c7d08578fdaafff3b4ba8ed70c95546cbe564981f3766051730: Status 404 returned error can't find the container with id 8c7cf2f569196c7d08578fdaafff3b4ba8ed70c95546cbe564981f3766051730 Nov 02 20:37:14.295412 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.295385 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx"] Nov 02 20:37:14.537070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.537040 4971 generic.go:356] "Generic (PLEG): container finished" podID="a02d7838-6668-43b5-b0d8-4e5af366edf1" containerID="aba37b744deaecfe6f274adca6819cd64d018045121794e71f96078a8bfd05b5" exitCode=0 Nov 02 20:37:14.537521 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.537118 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" event={"ID":"a02d7838-6668-43b5-b0d8-4e5af366edf1","Type":"ContainerDied","Data":"aba37b744deaecfe6f274adca6819cd64d018045121794e71f96078a8bfd05b5"} Nov 02 20:37:14.537521 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.537130 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" Nov 02 20:37:14.537521 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.537170 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv" event={"ID":"a02d7838-6668-43b5-b0d8-4e5af366edf1","Type":"ContainerDied","Data":"7d0c91cc2731bb7a06aa2abae54db61905b79792bfba94a597b4de2d281ad08c"} Nov 02 20:37:14.537521 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.537195 4971 scope.go:122] "RemoveContainer" containerID="aba37b744deaecfe6f274adca6819cd64d018045121794e71f96078a8bfd05b5" Nov 02 20:37:14.538946 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.538906 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx" event={"ID":"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a","Type":"ContainerStarted","Data":"8c7cf2f569196c7d08578fdaafff3b4ba8ed70c95546cbe564981f3766051730"} Nov 02 20:37:14.540488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.540464 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" Nov 02 20:37:14.540488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.540477 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp" event={"ID":"b0a8fe63-b184-4aa9-816b-eeb3c161ebbe","Type":"ContainerDied","Data":"7db14d4772df130ae7b32fcfbf6def1c0603feb1cd90446191f2563aab9781e6"} Nov 02 20:37:14.542728 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.542704 4971 generic.go:356] "Generic (PLEG): container finished" podID="d887a152-fb99-48f3-b702-779ce22c4e59" containerID="9bb50f82b71240c914fef16ba282343044bc995a659e444431f57404240117f0" exitCode=0 Nov 02 20:37:14.542867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.542794 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4" event={"ID":"d887a152-fb99-48f3-b702-779ce22c4e59","Type":"ContainerDied","Data":"9bb50f82b71240c914fef16ba282343044bc995a659e444431f57404240117f0"} Nov 02 20:37:14.544632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.544607 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" event={"ID":"817f7e35-0db0-493e-a3af-825e1818eb18","Type":"ContainerDied","Data":"c23460676d979890d53df175e60582a962856d64b45b348fd31bacc31642e16c"} Nov 02 20:37:14.544724 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.544614 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt" Nov 02 20:37:14.546276 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.546254 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" event={"ID":"e271f874-deb0-487e-a4bc-ee0f520bb501","Type":"ContainerDied","Data":"679444b3369cc92f89a87c833222dfd0c69bfe257f477e052a17c24b38f9d4bd"} Nov 02 20:37:14.546378 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.546313 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr" Nov 02 20:37:14.583601 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.583581 4971 scope.go:122] "RemoveContainer" containerID="aba37b744deaecfe6f274adca6819cd64d018045121794e71f96078a8bfd05b5" Nov 02 20:37:14.583846 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:14.583826 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba37b744deaecfe6f274adca6819cd64d018045121794e71f96078a8bfd05b5\": container with ID starting with aba37b744deaecfe6f274adca6819cd64d018045121794e71f96078a8bfd05b5 not found: ID does not exist" containerID="aba37b744deaecfe6f274adca6819cd64d018045121794e71f96078a8bfd05b5" Nov 02 20:37:14.583914 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.583854 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba37b744deaecfe6f274adca6819cd64d018045121794e71f96078a8bfd05b5"} err="failed to get container status \"aba37b744deaecfe6f274adca6819cd64d018045121794e71f96078a8bfd05b5\": rpc error: code = NotFound desc = could not find container \"aba37b744deaecfe6f274adca6819cd64d018045121794e71f96078a8bfd05b5\": container with ID starting with aba37b744deaecfe6f274adca6819cd64d018045121794e71f96078a8bfd05b5 not found: ID does not exist" Nov 02 20:37:14.583914 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.583870 4971 scope.go:122] "RemoveContainer" containerID="cf5042cffb0b6ec65d73a4cb8b8697be704113ffebdd1b558d161b99ad6d460c" Nov 02 20:37:14.598976 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.598958 4971 scope.go:122] "RemoveContainer" containerID="3052b3b7f6606fb2fd26bc1c17f042031c0fbcd9aa74be8ef86a3987798375eb" Nov 02 20:37:14.609125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.609077 4971 scope.go:122] "RemoveContainer" containerID="03b86e5c7290400c2b5186ef67d2287dda37bc042fe5423787f9086982991b46" Nov 02 20:37:14.956945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.956913 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv"] Nov 02 20:37:14.961655 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:14.961630 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-xx5bv"] Nov 02 20:37:15.064990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.064962 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt"] Nov 02 20:37:15.069588 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.069567 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gr2jt"] Nov 02 20:37:15.172209 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.172182 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp"] Nov 02 20:37:15.177879 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.177856 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-g69pp"] Nov 02 20:37:15.281685 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.281661 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr"] Nov 02 20:37:15.286102 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.286070 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c4bhr"] Nov 02 20:37:15.560840 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.560749 4971 generic.go:356] "Generic (PLEG): container finished" podID="240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a" containerID="f71d2e66e5a819055c640a388c71f20cb49fb642a2e004df2f7e5c87a4a7c249" exitCode=0 Nov 02 20:37:15.561304 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.560832 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx" event={"ID":"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a","Type":"ContainerDied","Data":"f71d2e66e5a819055c640a388c71f20cb49fb642a2e004df2f7e5c87a4a7c249"} Nov 02 20:37:15.735122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.735101 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4" Nov 02 20:37:15.757029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.757004 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7322-509722"] Nov 02 20:37:15.757653 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.757639 4971 cpu_manager.go:414] "containerMap: removing container" podUID="e271f874-deb0-487e-a4bc-ee0f520bb501" containerName="istio-proxy" Nov 02 20:37:15.757699 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.757655 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e271f874-deb0-487e-a4bc-ee0f520bb501" containerName="istio-proxy" Nov 02 20:37:15.757699 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.757675 4971 cpu_manager.go:414] "containerMap: removing container" podUID="a02d7838-6668-43b5-b0d8-4e5af366edf1" containerName="istio-proxy" Nov 02 20:37:15.757699 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.757681 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02d7838-6668-43b5-b0d8-4e5af366edf1" containerName="istio-proxy" Nov 02 20:37:15.757789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.757707 4971 cpu_manager.go:414] "containerMap: removing container" podUID="817f7e35-0db0-493e-a3af-825e1818eb18" containerName="istio-proxy" Nov 02 20:37:15.757789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.757712 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="817f7e35-0db0-493e-a3af-825e1818eb18" containerName="istio-proxy" Nov 02 20:37:15.757789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.757717 4971 cpu_manager.go:414] "containerMap: removing container" podUID="b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" containerName="istio-proxy" Nov 02 20:37:15.757789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.757722 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" containerName="istio-proxy" Nov 02 20:37:15.757789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.757736 4971 cpu_manager.go:414] "containerMap: removing container" podUID="d887a152-fb99-48f3-b702-779ce22c4e59" containerName="revoke-keys" Nov 02 20:37:15.757789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.757741 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d887a152-fb99-48f3-b702-779ce22c4e59" containerName="revoke-keys" Nov 02 20:37:15.758019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.757843 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="817f7e35-0db0-493e-a3af-825e1818eb18" containerName="istio-proxy" Nov 02 20:37:15.758019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.757852 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" containerName="istio-proxy" Nov 02 20:37:15.758019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.757860 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="d887a152-fb99-48f3-b702-779ce22c4e59" containerName="revoke-keys" Nov 02 20:37:15.758019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.757867 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="e271f874-deb0-487e-a4bc-ee0f520bb501" containerName="istio-proxy" Nov 02 20:37:15.758019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.757874 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02d7838-6668-43b5-b0d8-4e5af366edf1" containerName="istio-proxy" Nov 02 20:37:15.761411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.761397 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-509722" Nov 02 20:37:15.782009 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.781987 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d887a152-fb99-48f3-b702-779ce22c4e59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d887a152-fb99-48f3-b702-779ce22c4e59-maas-api-openshift-ca\") pod \"d887a152-fb99-48f3-b702-779ce22c4e59\" (UID: \"d887a152-fb99-48f3-b702-779ce22c4e59\") " Nov 02 20:37:15.782153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.782041 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d887a152-fb99-48f3-b702-779ce22c4e59-kube-api-access-4kn78\" (UniqueName: \"kubernetes.io/projected/d887a152-fb99-48f3-b702-779ce22c4e59-kube-api-access-4kn78\") pod \"d887a152-fb99-48f3-b702-779ce22c4e59\" (UID: \"d887a152-fb99-48f3-b702-779ce22c4e59\") " Nov 02 20:37:15.782259 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.782243 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d887a152-fb99-48f3-b702-779ce22c4e59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d887a152-fb99-48f3-b702-779ce22c4e59-maas-api-xks-ca\") pod \"d887a152-fb99-48f3-b702-779ce22c4e59\" (UID: \"d887a152-fb99-48f3-b702-779ce22c4e59\") " Nov 02 20:37:15.782337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.782317 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d887a152-fb99-48f3-b702-779ce22c4e59-maas-api-openshift-ca" pod "d887a152-fb99-48f3-b702-779ce22c4e59" (UID: "d887a152-fb99-48f3-b702-779ce22c4e59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:15.782506 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.782491 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xmv6\" (UniqueName: \"kubernetes.io/projected/04b4179b-0660-4bda-bc78-2d5ee0d88ae9-kube-api-access-8xmv6\") pod \"test-curl-7322-509722\" (UID: \"04b4179b-0660-4bda-bc78-2d5ee0d88ae9\") " pod="opendatahub/test-curl-7322-509722" Nov 02 20:37:15.782755 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.782739 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d887a152-fb99-48f3-b702-779ce22c4e59-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:15.784395 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.784369 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d887a152-fb99-48f3-b702-779ce22c4e59-kube-api-access-4kn78" pod "d887a152-fb99-48f3-b702-779ce22c4e59" (UID: "d887a152-fb99-48f3-b702-779ce22c4e59"). InnerVolumeSpecName "kube-api-access-4kn78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:15.784489 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.784470 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d887a152-fb99-48f3-b702-779ce22c4e59-maas-api-xks-ca" pod "d887a152-fb99-48f3-b702-779ce22c4e59" (UID: "d887a152-fb99-48f3-b702-779ce22c4e59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:15.810633 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.810613 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cmjwk\"" Nov 02 20:37:15.813597 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.813539 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-509722"] Nov 02 20:37:15.871854 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.871808 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="817f7e35-0db0-493e-a3af-825e1818eb18" path="/var/lib/kubelet/pods/817f7e35-0db0-493e-a3af-825e1818eb18/volumes" Nov 02 20:37:15.872588 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.872563 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a02d7838-6668-43b5-b0d8-4e5af366edf1" path="/var/lib/kubelet/pods/a02d7838-6668-43b5-b0d8-4e5af366edf1/volumes" Nov 02 20:37:15.873250 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.873227 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a8fe63-b184-4aa9-816b-eeb3c161ebbe" path="/var/lib/kubelet/pods/b0a8fe63-b184-4aa9-816b-eeb3c161ebbe/volumes" Nov 02 20:37:15.874410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.874386 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e271f874-deb0-487e-a4bc-ee0f520bb501" path="/var/lib/kubelet/pods/e271f874-deb0-487e-a4bc-ee0f520bb501/volumes" Nov 02 20:37:15.888100 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.884719 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xmv6\" (UniqueName: \"kubernetes.io/projected/04b4179b-0660-4bda-bc78-2d5ee0d88ae9-kube-api-access-8xmv6\") pod \"test-curl-7322-509722\" (UID: \"04b4179b-0660-4bda-bc78-2d5ee0d88ae9\") " pod="opendatahub/test-curl-7322-509722" Nov 02 20:37:15.888100 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.884936 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d887a152-fb99-48f3-b702-779ce22c4e59-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:15.888100 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.884964 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kn78\" (UniqueName: \"kubernetes.io/projected/d887a152-fb99-48f3-b702-779ce22c4e59-kube-api-access-4kn78\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:15.942662 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:15.942638 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xmv6\" (UniqueName: \"kubernetes.io/projected/04b4179b-0660-4bda-bc78-2d5ee0d88ae9-kube-api-access-8xmv6\") pod \"test-curl-7322-509722\" (UID: \"04b4179b-0660-4bda-bc78-2d5ee0d88ae9\") " pod="opendatahub/test-curl-7322-509722" Nov 02 20:37:16.074219 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.074149 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-509722" Nov 02 20:37:16.447373 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:16.447340 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04b4179b_0660_4bda_bc78_2d5ee0d88ae9.slice/crio-85ae9cdc7cae7f732d8de30f18f0c4a1239d864aac0302220f927ff2ff0e6f95 WatchSource:0}: Error finding container 85ae9cdc7cae7f732d8de30f18f0c4a1239d864aac0302220f927ff2ff0e6f95: Status 404 returned error can't find the container with id 85ae9cdc7cae7f732d8de30f18f0c4a1239d864aac0302220f927ff2ff0e6f95 Nov 02 20:37:16.448416 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.448390 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-509722"] Nov 02 20:37:16.449370 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.449353 4971 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:37:16.571932 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.571903 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-509722" event={"ID":"04b4179b-0660-4bda-bc78-2d5ee0d88ae9","Type":"ContainerStarted","Data":"85ae9cdc7cae7f732d8de30f18f0c4a1239d864aac0302220f927ff2ff0e6f95"} Nov 02 20:37:16.573544 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.573522 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4" Nov 02 20:37:16.573644 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.573535 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4" event={"ID":"d887a152-fb99-48f3-b702-779ce22c4e59","Type":"ContainerDied","Data":"9cfdbdfdd8aca7d49da537f7023c10b1571144f84b9b33cc93b4e8d52b98e944"} Nov 02 20:37:16.573644 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.573618 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cfdbdfdd8aca7d49da537f7023c10b1571144f84b9b33cc93b4e8d52b98e944" Nov 02 20:37:16.738959 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.738936 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx" Nov 02 20:37:16.795465 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.795440 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-maas-api-openshift-ca\") pod \"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a\" (UID: \"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a\") " Nov 02 20:37:16.795613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.795560 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-maas-api-xks-ca\") pod \"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a\" (UID: \"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a\") " Nov 02 20:37:16.795713 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.795685 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-kube-api-access-xthx6\" (UniqueName: \"kubernetes.io/projected/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-kube-api-access-xthx6\") pod \"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a\" (UID: \"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a\") " Nov 02 20:37:16.795842 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.795818 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-maas-api-openshift-ca" pod "240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a" (UID: "240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:16.796142 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.796122 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:16.797789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.797767 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-maas-api-xks-ca" pod "240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a" (UID: "240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:16.797937 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.797920 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-kube-api-access-xthx6" pod "240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a" (UID: "240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a"). InnerVolumeSpecName "kube-api-access-xthx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:16.897864 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.897835 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xthx6\" (UniqueName: \"kubernetes.io/projected/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-kube-api-access-xthx6\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:16.897998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:16.897871 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:17.584295 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:17.584261 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx" event={"ID":"240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a","Type":"ContainerDied","Data":"8c7cf2f569196c7d08578fdaafff3b4ba8ed70c95546cbe564981f3766051730"} Nov 02 20:37:17.584295 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:17.584293 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx" Nov 02 20:37:17.584697 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:17.584317 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c7cf2f569196c7d08578fdaafff3b4ba8ed70c95546cbe564981f3766051730" Nov 02 20:37:17.878371 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:17.878347 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc"] Nov 02 20:37:17.879118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:17.879094 4971 cpu_manager.go:414] "containerMap: removing container" podUID="240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a" containerName="revoke-keys" Nov 02 20:37:17.879118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:17.879116 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a" containerName="revoke-keys" Nov 02 20:37:17.879272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:17.879261 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a" containerName="revoke-keys" Nov 02 20:37:17.882982 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:17.882965 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc" Nov 02 20:37:17.910033 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:17.909611 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq5g9\" (UniqueName: \"kubernetes.io/projected/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-kube-api-access-xq5g9\") pod \"maas-api-revoke-sub-077f127e-schzc\" (UID: \"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc" Nov 02 20:37:17.910033 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:17.909771 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-077f127e-schzc\" (UID: \"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc" Nov 02 20:37:17.910033 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:17.909957 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-077f127e-schzc\" (UID: \"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc" Nov 02 20:37:17.932261 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:17.932238 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:37:17.932504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:17.932485 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:37:17.974048 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:17.974019 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc"] Nov 02 20:37:18.011644 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.011613 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-077f127e-schzc\" (UID: \"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc" Nov 02 20:37:18.011810 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.011727 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq5g9\" (UniqueName: \"kubernetes.io/projected/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-kube-api-access-xq5g9\") pod \"maas-api-revoke-sub-077f127e-schzc\" (UID: \"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc" Nov 02 20:37:18.011888 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.011819 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-077f127e-schzc\" (UID: \"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc" Nov 02 20:37:18.012666 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.012639 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-077f127e-schzc\" (UID: \"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc" Nov 02 20:37:18.014349 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.014320 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-077f127e-schzc\" (UID: \"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc" Nov 02 20:37:18.049238 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.049217 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq"] Nov 02 20:37:18.054062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.054043 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm"] Nov 02 20:37:18.058144 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.058130 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46b8e40d-h7bqq"] Nov 02 20:37:18.061938 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.061920 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59586e36-q7kbm"] Nov 02 20:37:18.069559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.069539 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq5g9\" (UniqueName: \"kubernetes.io/projected/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-kube-api-access-xq5g9\") pod \"maas-api-revoke-sub-077f127e-schzc\" (UID: \"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc" Nov 02 20:37:18.149915 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.149888 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg"] Nov 02 20:37:18.154622 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.154603 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" Nov 02 20:37:18.196603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.196579 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc" Nov 02 20:37:18.211618 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.211594 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg"] Nov 02 20:37:18.214405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.214379 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7zqw\" (UniqueName: \"kubernetes.io/projected/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-kube-api-access-h7zqw\") pod \"maas-api-revoke-sub-a7bd60a1-wbpfg\" (UID: \"5c4cf0f5-1c2f-411b-b52c-441569c8e62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" Nov 02 20:37:18.214520 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.214426 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7bd60a1-wbpfg\" (UID: \"5c4cf0f5-1c2f-411b-b52c-441569c8e62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" Nov 02 20:37:18.214612 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.214595 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7bd60a1-wbpfg\" (UID: \"5c4cf0f5-1c2f-411b-b52c-441569c8e62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" Nov 02 20:37:18.315686 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.315654 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7zqw\" (UniqueName: \"kubernetes.io/projected/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-kube-api-access-h7zqw\") pod \"maas-api-revoke-sub-a7bd60a1-wbpfg\" (UID: \"5c4cf0f5-1c2f-411b-b52c-441569c8e62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" Nov 02 20:37:18.315851 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.315704 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7bd60a1-wbpfg\" (UID: \"5c4cf0f5-1c2f-411b-b52c-441569c8e62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" Nov 02 20:37:18.315936 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.315891 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7bd60a1-wbpfg\" (UID: \"5c4cf0f5-1c2f-411b-b52c-441569c8e62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" Nov 02 20:37:18.316631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.316572 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7bd60a1-wbpfg\" (UID: \"5c4cf0f5-1c2f-411b-b52c-441569c8e62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" Nov 02 20:37:18.318492 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.318466 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7bd60a1-wbpfg\" (UID: \"5c4cf0f5-1c2f-411b-b52c-441569c8e62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" Nov 02 20:37:18.373354 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.373326 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7zqw\" (UniqueName: \"kubernetes.io/projected/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-kube-api-access-h7zqw\") pod \"maas-api-revoke-sub-a7bd60a1-wbpfg\" (UID: \"5c4cf0f5-1c2f-411b-b52c-441569c8e62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" Nov 02 20:37:18.470060 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.470007 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" Nov 02 20:37:18.594252 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.594222 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc"] Nov 02 20:37:18.602849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.602818 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-509722" event={"ID":"04b4179b-0660-4bda-bc78-2d5ee0d88ae9","Type":"ContainerStarted","Data":"d7daefa4aa3f9398d1ef60b217d5c4205e19ae9d1442ed80087c2ebada1a3cc1"} Nov 02 20:37:18.712769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.712713 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7322-509722" podStartSLOduration=2.392643325 podStartE2EDuration="3.712698406s" totalImagesPullingTime="1.320055081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:18.709057615 +0000 UTC m=+1366.998714145" watchObservedRunningTime="2026-11-02 20:37:18.712698406 +0000 UTC m=+1367.002354931" Nov 02 20:37:18.718388 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.718356 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:18.847354 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:18.847326 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg"] Nov 02 20:37:18.852584 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:18.852561 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4cf0f5_1c2f_411b_b52c_441569c8e62e.slice/crio-bd4fe2215e170b94cb4a762b4ba9af74f7ee706da0c036c28a8ad5e757ac83b4 WatchSource:0}: Error finding container bd4fe2215e170b94cb4a762b4ba9af74f7ee706da0c036c28a8ad5e757ac83b4: Status 404 returned error can't find the container with id bd4fe2215e170b94cb4a762b4ba9af74f7ee706da0c036c28a8ad5e757ac83b4 Nov 02 20:37:19.612769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:19.612740 4971 generic.go:356] "Generic (PLEG): container finished" podID="ad3ecf0f-5677-4afa-884b-a48f7ccd99f5" containerID="679f71de21902e84040206a438e42c6156fd2cb60c0bf42b6edf1793891e5244" exitCode=0 Nov 02 20:37:19.613185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:19.612819 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc" event={"ID":"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5","Type":"ContainerDied","Data":"679f71de21902e84040206a438e42c6156fd2cb60c0bf42b6edf1793891e5244"} Nov 02 20:37:19.613185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:19.612874 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc" event={"ID":"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5","Type":"ContainerStarted","Data":"e4d53f5b1d23e205f5565d33b72fe503439fbc6eeea915f2c2dbf683f478e642"} Nov 02 20:37:19.614766 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:19.614734 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" event={"ID":"5c4cf0f5-1c2f-411b-b52c-441569c8e62e","Type":"ContainerStarted","Data":"45f2d198b2342692e20c55e21228c9725065f98ffc92341f151574c6dc465122"} Nov 02 20:37:19.614873 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:19.614777 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" event={"ID":"5c4cf0f5-1c2f-411b-b52c-441569c8e62e","Type":"ContainerStarted","Data":"bd4fe2215e170b94cb4a762b4ba9af74f7ee706da0c036c28a8ad5e757ac83b4"} Nov 02 20:37:19.835532 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:19.835440 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" podStartSLOduration=1.835426765 podStartE2EDuration="1.835426765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:19.830949206 +0000 UTC m=+1368.120605734" watchObservedRunningTime="2026-11-02 20:37:19.835426765 +0000 UTC m=+1368.125083292" Nov 02 20:37:19.868050 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:19.868022 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32fe9460-fed6-4e8d-8625-5915605902e0" path="/var/lib/kubelet/pods/32fe9460-fed6-4e8d-8625-5915605902e0/volumes" Nov 02 20:37:19.868505 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:19.868491 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cebecc27-e1dd-4b63-bec3-20dbba31e58b" path="/var/lib/kubelet/pods/cebecc27-e1dd-4b63-bec3-20dbba31e58b/volumes" Nov 02 20:37:20.124263 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.124234 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7322-509722"] Nov 02 20:37:20.130919 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.130890 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7322-509722"] Nov 02 20:37:20.273192 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.273160 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7322-cd6d2c"] Nov 02 20:37:20.273863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.273849 4971 cpu_manager.go:414] "containerMap: removing container" podUID="04b4179b-0660-4bda-bc78-2d5ee0d88ae9" containerName="test-curl-7322-509722" Nov 02 20:37:20.273917 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.273865 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b4179b-0660-4bda-bc78-2d5ee0d88ae9" containerName="test-curl-7322-509722" Nov 02 20:37:20.273993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.273984 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b4179b-0660-4bda-bc78-2d5ee0d88ae9" containerName="test-curl-7322-509722" Nov 02 20:37:20.277629 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.277612 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-cd6d2c" Nov 02 20:37:20.329623 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.329595 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-cd6d2c"] Nov 02 20:37:20.342159 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.342133 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk26b\" (UniqueName: \"kubernetes.io/projected/7a44d513-0188-4dc4-a6ef-e4214d30d9ef-kube-api-access-fk26b\") pod \"test-curl-7322-cd6d2c\" (UID: \"7a44d513-0188-4dc4-a6ef-e4214d30d9ef\") " pod="opendatahub/test-curl-7322-cd6d2c" Nov 02 20:37:20.444202 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.444133 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk26b\" (UniqueName: \"kubernetes.io/projected/7a44d513-0188-4dc4-a6ef-e4214d30d9ef-kube-api-access-fk26b\") pod \"test-curl-7322-cd6d2c\" (UID: \"7a44d513-0188-4dc4-a6ef-e4214d30d9ef\") " pod="opendatahub/test-curl-7322-cd6d2c" Nov 02 20:37:20.501793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.501762 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk26b\" (UniqueName: \"kubernetes.io/projected/7a44d513-0188-4dc4-a6ef-e4214d30d9ef-kube-api-access-fk26b\") pod \"test-curl-7322-cd6d2c\" (UID: \"7a44d513-0188-4dc4-a6ef-e4214d30d9ef\") " pod="opendatahub/test-curl-7322-cd6d2c" Nov 02 20:37:20.590129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.590104 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-cd6d2c" Nov 02 20:37:20.628353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.628320 4971 generic.go:356] "Generic (PLEG): container finished" podID="5c4cf0f5-1c2f-411b-b52c-441569c8e62e" containerID="45f2d198b2342692e20c55e21228c9725065f98ffc92341f151574c6dc465122" exitCode=0 Nov 02 20:37:20.628682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.628403 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" event={"ID":"5c4cf0f5-1c2f-411b-b52c-441569c8e62e","Type":"ContainerDied","Data":"45f2d198b2342692e20c55e21228c9725065f98ffc92341f151574c6dc465122"} Nov 02 20:37:20.628844 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.628819 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7322-509722" podUID="04b4179b-0660-4bda-bc78-2d5ee0d88ae9" containerName="test-curl-7322-509722" containerID="cri-o://d7daefa4aa3f9398d1ef60b217d5c4205e19ae9d1442ed80087c2ebada1a3cc1" gracePeriod=2 Nov 02 20:37:20.787781 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:20.787737 4971 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-509722\" is forbidden: User \"system:node:ip-10-0-140-120.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-120.us-west-2.compute.internal' and this object" podUID="04b4179b-0660-4bda-bc78-2d5ee0d88ae9" pod="opendatahub/test-curl-7322-509722" Nov 02 20:37:20.836888 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.836861 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc" Nov 02 20:37:20.885235 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:20.885201 4971 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-509722\" is forbidden: User \"system:node:ip-10-0-140-120.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-120.us-west-2.compute.internal' and this object" podUID="04b4179b-0660-4bda-bc78-2d5ee0d88ae9" pod="opendatahub/test-curl-7322-509722" Nov 02 20:37:20.952908 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.952884 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-maas-api-xks-ca\") pod \"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5\" (UID: \"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5\") " Nov 02 20:37:20.953040 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.952957 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-maas-api-openshift-ca\") pod \"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5\" (UID: \"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5\") " Nov 02 20:37:20.953162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.953143 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-kube-api-access-xq5g9\" (UniqueName: \"kubernetes.io/projected/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-kube-api-access-xq5g9\") pod \"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5\" (UID: \"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5\") " Nov 02 20:37:20.953377 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.953349 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-maas-api-openshift-ca" pod "ad3ecf0f-5677-4afa-884b-a48f7ccd99f5" (UID: "ad3ecf0f-5677-4afa-884b-a48f7ccd99f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:20.953674 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.953651 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:20.955271 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.955245 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-maas-api-xks-ca" pod "ad3ecf0f-5677-4afa-884b-a48f7ccd99f5" (UID: "ad3ecf0f-5677-4afa-884b-a48f7ccd99f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:20.955420 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.955399 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-kube-api-access-xq5g9" pod "ad3ecf0f-5677-4afa-884b-a48f7ccd99f5" (UID: "ad3ecf0f-5677-4afa-884b-a48f7ccd99f5"). InnerVolumeSpecName "kube-api-access-xq5g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:20.968938 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:20.968912 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a44d513_0188_4dc4_a6ef_e4214d30d9ef.slice/crio-22594eb0cba8f40a1a415ec72bc59035d7a228503aaf8e3ed2b710befd86f827 WatchSource:0}: Error finding container 22594eb0cba8f40a1a415ec72bc59035d7a228503aaf8e3ed2b710befd86f827: Status 404 returned error can't find the container with id 22594eb0cba8f40a1a415ec72bc59035d7a228503aaf8e3ed2b710befd86f827 Nov 02 20:37:20.970420 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:20.970398 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7322-cd6d2c"] Nov 02 20:37:21.055675 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.055645 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq5g9\" (UniqueName: \"kubernetes.io/projected/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-kube-api-access-xq5g9\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:21.055761 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.055685 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:21.438270 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.438232 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4"] Nov 02 20:37:21.438997 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.438976 4971 cpu_manager.go:414] "containerMap: removing container" podUID="ad3ecf0f-5677-4afa-884b-a48f7ccd99f5" containerName="revoke-keys" Nov 02 20:37:21.438997 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.438997 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3ecf0f-5677-4afa-884b-a48f7ccd99f5" containerName="revoke-keys" Nov 02 20:37:21.439230 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.439215 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3ecf0f-5677-4afa-884b-a48f7ccd99f5" containerName="revoke-keys" Nov 02 20:37:21.442947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.442930 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.491223 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:21.491195 4971 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-509722\" is forbidden: User \"system:node:ip-10-0-140-120.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-120.us-west-2.compute.internal' and this object" podUID="04b4179b-0660-4bda-bc78-2d5ee0d88ae9" pod="opendatahub/test-curl-7322-509722" Nov 02 20:37:21.561326 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.561287 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.561474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.561340 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.561474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.561390 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.561474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.561450 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.561669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.561553 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.561669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.561648 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.561946 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.561917 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4r55\" (UniqueName: \"kubernetes.io/projected/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-kube-api-access-f4r55\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.562067 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.561982 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.562067 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.562032 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.569905 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.569880 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4"] Nov 02 20:37:21.639117 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.639053 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc" Nov 02 20:37:21.639117 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.639099 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc" event={"ID":"ad3ecf0f-5677-4afa-884b-a48f7ccd99f5","Type":"ContainerDied","Data":"e4d53f5b1d23e205f5565d33b72fe503439fbc6eeea915f2c2dbf683f478e642"} Nov 02 20:37:21.639575 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.639148 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4d53f5b1d23e205f5565d33b72fe503439fbc6eeea915f2c2dbf683f478e642" Nov 02 20:37:21.640950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.640921 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-cd6d2c" event={"ID":"7a44d513-0188-4dc4-a6ef-e4214d30d9ef","Type":"ContainerStarted","Data":"669bdb6614b05221584ea9d2241ee32b0d056f8be2ac997cd818371e48ba2924"} Nov 02 20:37:21.641052 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.640963 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7322-cd6d2c" event={"ID":"7a44d513-0188-4dc4-a6ef-e4214d30d9ef","Type":"ContainerStarted","Data":"22594eb0cba8f40a1a415ec72bc59035d7a228503aaf8e3ed2b710befd86f827"} Nov 02 20:37:21.663010 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.662980 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.663156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.663072 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.663156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.663134 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.663282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.663180 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.663282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.663236 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.663385 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.663332 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.663444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.663417 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.663550 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.663447 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.663550 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.663473 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4r55\" (UniqueName: \"kubernetes.io/projected/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-kube-api-access-f4r55\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.663550 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.663526 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.663704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.663653 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.663704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.663691 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.663871 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.663731 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.663871 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.663782 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.665841 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.665822 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.666077 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.666054 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.722133 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.722065 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.722739 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.722687 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4r55\" (UniqueName: \"kubernetes.io/projected/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-kube-api-access-f4r55\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.754261 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.754202 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7322-cd6d2c" podStartSLOduration=1.75417677 podStartE2EDuration="1.75417677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:21.747310286 +0000 UTC m=+1370.036966814" watchObservedRunningTime="2026-11-02 20:37:21.75417677 +0000 UTC m=+1370.043833299" Nov 02 20:37:21.757532 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.757506 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:21.786945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.786919 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" Nov 02 20:37:21.795516 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:21.795488 4971 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-509722\" is forbidden: User \"system:node:ip-10-0-140-120.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-120.us-west-2.compute.internal' and this object" podUID="04b4179b-0660-4bda-bc78-2d5ee0d88ae9" pod="opendatahub/test-curl-7322-509722" Nov 02 20:37:21.865309 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.865269 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-kube-api-access-h7zqw\" (UniqueName: \"kubernetes.io/projected/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-kube-api-access-h7zqw\") pod \"5c4cf0f5-1c2f-411b-b52c-441569c8e62e\" (UID: \"5c4cf0f5-1c2f-411b-b52c-441569c8e62e\") " Nov 02 20:37:21.865445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.865319 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-maas-api-openshift-ca\") pod \"5c4cf0f5-1c2f-411b-b52c-441569c8e62e\" (UID: \"5c4cf0f5-1c2f-411b-b52c-441569c8e62e\") " Nov 02 20:37:21.865526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.865494 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-maas-api-xks-ca\") pod \"5c4cf0f5-1c2f-411b-b52c-441569c8e62e\" (UID: \"5c4cf0f5-1c2f-411b-b52c-441569c8e62e\") " Nov 02 20:37:21.865781 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.865734 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-maas-api-openshift-ca" pod "5c4cf0f5-1c2f-411b-b52c-441569c8e62e" (UID: "5c4cf0f5-1c2f-411b-b52c-441569c8e62e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:21.866185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.866152 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:21.869306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.869281 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-kube-api-access-h7zqw" pod "5c4cf0f5-1c2f-411b-b52c-441569c8e62e" (UID: "5c4cf0f5-1c2f-411b-b52c-441569c8e62e"). InnerVolumeSpecName "kube-api-access-h7zqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:21.869704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.869682 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-maas-api-xks-ca" pod "5c4cf0f5-1c2f-411b-b52c-441569c8e62e" (UID: "5c4cf0f5-1c2f-411b-b52c-441569c8e62e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:21.893290 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:21.893259 4971 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-509722\" is forbidden: User \"system:node:ip-10-0-140-120.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-120.us-west-2.compute.internal' and this object" podUID="04b4179b-0660-4bda-bc78-2d5ee0d88ae9" pod="opendatahub/test-curl-7322-509722" Nov 02 20:37:21.967527 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.967452 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:21.967527 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:21.967480 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7zqw\" (UniqueName: \"kubernetes.io/projected/5c4cf0f5-1c2f-411b-b52c-441569c8e62e-kube-api-access-h7zqw\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:22.185375 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:22.185344 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcede60_ff17_4d6a_b3ab_65dea8834b5e.slice/crio-4c2b87df56bb605d0a6492a15ea810ed8492f9432cc299e99f7933516095c2b8 WatchSource:0}: Error finding container 4c2b87df56bb605d0a6492a15ea810ed8492f9432cc299e99f7933516095c2b8: Status 404 returned error can't find the container with id 4c2b87df56bb605d0a6492a15ea810ed8492f9432cc299e99f7933516095c2b8 Nov 02 20:37:22.186793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.186768 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4"] Nov 02 20:37:22.188281 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.188236 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:22.188355 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.188319 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:22.188407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.188361 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:22.399312 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.399279 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:22.648267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.648229 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:22.657890 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.655212 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" event={"ID":"5c4cf0f5-1c2f-411b-b52c-441569c8e62e","Type":"ContainerDied","Data":"bd4fe2215e170b94cb4a762b4ba9af74f7ee706da0c036c28a8ad5e757ac83b4"} Nov 02 20:37:22.657890 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.655275 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd4fe2215e170b94cb4a762b4ba9af74f7ee706da0c036c28a8ad5e757ac83b4" Nov 02 20:37:22.657890 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.655456 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg" Nov 02 20:37:22.658281 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.658254 4971 generic.go:356] "Generic (PLEG): container finished" podID="04b4179b-0660-4bda-bc78-2d5ee0d88ae9" containerID="d7daefa4aa3f9398d1ef60b217d5c4205e19ae9d1442ed80087c2ebada1a3cc1" exitCode=137 Nov 02 20:37:22.660592 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.660567 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" event={"ID":"dfcede60-ff17-4d6a-b3ab-65dea8834b5e","Type":"ContainerStarted","Data":"c51eee1beefea6d7d5e4ab4838f8302c202dcce84b1f9f51fdff5b5797cb84b8"} Nov 02 20:37:22.660684 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.660626 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" event={"ID":"dfcede60-ff17-4d6a-b3ab-65dea8834b5e","Type":"ContainerStarted","Data":"4c2b87df56bb605d0a6492a15ea810ed8492f9432cc299e99f7933516095c2b8"} Nov 02 20:37:22.758215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.758074 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:22.780012 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.779948 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" podStartSLOduration=1.7799296390000001 podStartE2EDuration="1.779929639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:22.775024404 +0000 UTC m=+1371.064680945" watchObservedRunningTime="2026-11-02 20:37:22.779929639 +0000 UTC m=+1371.069586185" Nov 02 20:37:22.850542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.850519 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-509722" Nov 02 20:37:22.904145 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:22.898805 4971 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-509722\" is forbidden: User \"system:node:ip-10-0-140-120.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-120.us-west-2.compute.internal' and this object" podUID="04b4179b-0660-4bda-bc78-2d5ee0d88ae9" pod="opendatahub/test-curl-7322-509722" Nov 02 20:37:22.986614 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.986585 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b4179b-0660-4bda-bc78-2d5ee0d88ae9-kube-api-access-8xmv6\" (UniqueName: \"kubernetes.io/projected/04b4179b-0660-4bda-bc78-2d5ee0d88ae9-kube-api-access-8xmv6\") pod \"04b4179b-0660-4bda-bc78-2d5ee0d88ae9\" (UID: \"04b4179b-0660-4bda-bc78-2d5ee0d88ae9\") " Nov 02 20:37:22.989511 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:22.989482 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b4179b-0660-4bda-bc78-2d5ee0d88ae9-kube-api-access-8xmv6" pod "04b4179b-0660-4bda-bc78-2d5ee0d88ae9" (UID: "04b4179b-0660-4bda-bc78-2d5ee0d88ae9"). InnerVolumeSpecName "kube-api-access-8xmv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:23.088590 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.088555 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xmv6\" (UniqueName: \"kubernetes.io/projected/04b4179b-0660-4bda-bc78-2d5ee0d88ae9-kube-api-access-8xmv6\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:23.209732 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.209639 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:23.279676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.279638 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7322-cd6d2c"] Nov 02 20:37:23.287187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.287149 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7322-cd6d2c"] Nov 02 20:37:23.642171 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.637102 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh"] Nov 02 20:37:23.642171 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.638159 4971 cpu_manager.go:414] "containerMap: removing container" podUID="5c4cf0f5-1c2f-411b-b52c-441569c8e62e" containerName="revoke-keys" Nov 02 20:37:23.642171 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.638179 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c4cf0f5-1c2f-411b-b52c-441569c8e62e" containerName="revoke-keys" Nov 02 20:37:23.642171 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.638261 4971 cpu_manager.go:414] "containerMap: removing container" podUID="7a44d513-0188-4dc4-a6ef-e4214d30d9ef" containerName="test-curl-7322-cd6d2c" Nov 02 20:37:23.642171 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.638271 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a44d513-0188-4dc4-a6ef-e4214d30d9ef" containerName="test-curl-7322-cd6d2c" Nov 02 20:37:23.642171 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.638440 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a44d513-0188-4dc4-a6ef-e4214d30d9ef" containerName="test-curl-7322-cd6d2c" Nov 02 20:37:23.642171 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.638460 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c4cf0f5-1c2f-411b-b52c-441569c8e62e" containerName="revoke-keys" Nov 02 20:37:23.654153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.643337 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" Nov 02 20:37:23.654153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.653829 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:23.671942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.671352 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-509722" Nov 02 20:37:23.671942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.671397 4971 scope.go:122] "RemoveContainer" containerID="d7daefa4aa3f9398d1ef60b217d5c4205e19ae9d1442ed80087c2ebada1a3cc1" Nov 02 20:37:23.671942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.671525 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7322-cd6d2c" podUID="7a44d513-0188-4dc4-a6ef-e4214d30d9ef" containerName="test-curl-7322-cd6d2c" containerID="cri-o://669bdb6614b05221584ea9d2241ee32b0d056f8be2ac997cd818371e48ba2924" gracePeriod=2 Nov 02 20:37:23.690391 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.690361 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh"] Nov 02 20:37:23.697902 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.697872 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2910c32a-6f07-47fd-8b35-bade9e68d34d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c393931-5fmzh\" (UID: \"2910c32a-6f07-47fd-8b35-bade9e68d34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" Nov 02 20:37:23.698027 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.697945 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjpn5\" (UniqueName: \"kubernetes.io/projected/2910c32a-6f07-47fd-8b35-bade9e68d34d-kube-api-access-vjpn5\") pod \"maas-api-revoke-sub-9c393931-5fmzh\" (UID: \"2910c32a-6f07-47fd-8b35-bade9e68d34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" Nov 02 20:37:23.698152 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.698072 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2910c32a-6f07-47fd-8b35-bade9e68d34d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c393931-5fmzh\" (UID: \"2910c32a-6f07-47fd-8b35-bade9e68d34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" Nov 02 20:37:23.703885 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.703860 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:37:23.704036 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.704021 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:37:23.759677 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.759645 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:37:23.759831 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.759699 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" podUID="dfcede60-ff17-4d6a-b3ab-65dea8834b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 20:37:23.799231 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.799195 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2910c32a-6f07-47fd-8b35-bade9e68d34d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c393931-5fmzh\" (UID: \"2910c32a-6f07-47fd-8b35-bade9e68d34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" Nov 02 20:37:23.799370 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.799275 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjpn5\" (UniqueName: \"kubernetes.io/projected/2910c32a-6f07-47fd-8b35-bade9e68d34d-kube-api-access-vjpn5\") pod \"maas-api-revoke-sub-9c393931-5fmzh\" (UID: \"2910c32a-6f07-47fd-8b35-bade9e68d34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" Nov 02 20:37:23.799475 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.799445 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2910c32a-6f07-47fd-8b35-bade9e68d34d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c393931-5fmzh\" (UID: \"2910c32a-6f07-47fd-8b35-bade9e68d34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" Nov 02 20:37:23.799902 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.799883 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2910c32a-6f07-47fd-8b35-bade9e68d34d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c393931-5fmzh\" (UID: \"2910c32a-6f07-47fd-8b35-bade9e68d34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" Nov 02 20:37:23.801936 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.801916 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2910c32a-6f07-47fd-8b35-bade9e68d34d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c393931-5fmzh\" (UID: \"2910c32a-6f07-47fd-8b35-bade9e68d34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" Nov 02 20:37:23.809949 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:23.809923 4971 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-509722\" is forbidden: User \"system:node:ip-10-0-140-120.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-120.us-west-2.compute.internal' and this object" podUID="04b4179b-0660-4bda-bc78-2d5ee0d88ae9" pod="opendatahub/test-curl-7322-509722" Nov 02 20:37:23.857201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.857171 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjpn5\" (UniqueName: \"kubernetes.io/projected/2910c32a-6f07-47fd-8b35-bade9e68d34d-kube-api-access-vjpn5\") pod \"maas-api-revoke-sub-9c393931-5fmzh\" (UID: \"2910c32a-6f07-47fd-8b35-bade9e68d34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" Nov 02 20:37:23.857979 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:23.857945 4971 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-509722\" is forbidden: User \"system:node:ip-10-0-140-120.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-120.us-west-2.compute.internal' and this object" podUID="04b4179b-0660-4bda-bc78-2d5ee0d88ae9" pod="opendatahub/test-curl-7322-509722" Nov 02 20:37:23.869228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.869200 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04b4179b-0660-4bda-bc78-2d5ee0d88ae9" path="/var/lib/kubelet/pods/04b4179b-0660-4bda-bc78-2d5ee0d88ae9/volumes" Nov 02 20:37:23.906195 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:23.906124 4971 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-cd6d2c\" is forbidden: User \"system:node:ip-10-0-140-120.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-120.us-west-2.compute.internal' and this object" podUID="7a44d513-0188-4dc4-a6ef-e4214d30d9ef" pod="opendatahub/test-curl-7322-cd6d2c" Nov 02 20:37:23.965628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:23.965601 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" Nov 02 20:37:24.156781 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.156700 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v"] Nov 02 20:37:24.165299 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.165275 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" Nov 02 20:37:24.190024 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.190000 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v"] Nov 02 20:37:24.211250 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.211224 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:24.306966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.306926 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2rrl\" (UniqueName: \"kubernetes.io/projected/a173d1e4-942e-4773-b9a4-ae854a61072d-kube-api-access-q2rrl\") pod \"maas-api-revoke-sub-694b51ab-lhm4v\" (UID: \"a173d1e4-942e-4773-b9a4-ae854a61072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" Nov 02 20:37:24.306966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.306981 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a173d1e4-942e-4773-b9a4-ae854a61072d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-694b51ab-lhm4v\" (UID: \"a173d1e4-942e-4773-b9a4-ae854a61072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" Nov 02 20:37:24.307205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.307163 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a173d1e4-942e-4773-b9a4-ae854a61072d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-694b51ab-lhm4v\" (UID: \"a173d1e4-942e-4773-b9a4-ae854a61072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" Nov 02 20:37:24.381274 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:24.380596 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2910c32a_6f07_47fd_8b35_bade9e68d34d.slice/crio-f5f220a9d4bc3c9cf2fa9553369b2f3536a66a89183ca9658c176d63dd5b3b36 WatchSource:0}: Error finding container f5f220a9d4bc3c9cf2fa9553369b2f3536a66a89183ca9658c176d63dd5b3b36: Status 404 returned error can't find the container with id f5f220a9d4bc3c9cf2fa9553369b2f3536a66a89183ca9658c176d63dd5b3b36 Nov 02 20:37:24.383596 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.383571 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh"] Nov 02 20:37:24.408752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.408687 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a173d1e4-942e-4773-b9a4-ae854a61072d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-694b51ab-lhm4v\" (UID: \"a173d1e4-942e-4773-b9a4-ae854a61072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" Nov 02 20:37:24.408869 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.408818 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a173d1e4-942e-4773-b9a4-ae854a61072d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-694b51ab-lhm4v\" (UID: \"a173d1e4-942e-4773-b9a4-ae854a61072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" Nov 02 20:37:24.408941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.408916 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2rrl\" (UniqueName: \"kubernetes.io/projected/a173d1e4-942e-4773-b9a4-ae854a61072d-kube-api-access-q2rrl\") pod \"maas-api-revoke-sub-694b51ab-lhm4v\" (UID: \"a173d1e4-942e-4773-b9a4-ae854a61072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" Nov 02 20:37:24.409642 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.409616 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a173d1e4-942e-4773-b9a4-ae854a61072d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-694b51ab-lhm4v\" (UID: \"a173d1e4-942e-4773-b9a4-ae854a61072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" Nov 02 20:37:24.411332 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.411309 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a173d1e4-942e-4773-b9a4-ae854a61072d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-694b51ab-lhm4v\" (UID: \"a173d1e4-942e-4773-b9a4-ae854a61072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" Nov 02 20:37:24.466947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.466921 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2rrl\" (UniqueName: \"kubernetes.io/projected/a173d1e4-942e-4773-b9a4-ae854a61072d-kube-api-access-q2rrl\") pod \"maas-api-revoke-sub-694b51ab-lhm4v\" (UID: \"a173d1e4-942e-4773-b9a4-ae854a61072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" Nov 02 20:37:24.484869 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.484845 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" Nov 02 20:37:24.660180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.658294 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9"] Nov 02 20:37:24.667153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.667114 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.695191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.695158 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" event={"ID":"2910c32a-6f07-47fd-8b35-bade9e68d34d","Type":"ContainerStarted","Data":"f5f220a9d4bc3c9cf2fa9553369b2f3536a66a89183ca9658c176d63dd5b3b36"} Nov 02 20:37:24.706273 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.706246 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9"] Nov 02 20:37:24.830777 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.830739 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2b3e20d8-openshift-default-dockercfg-gm8wh\"" Nov 02 20:37:24.880108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.865980 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:24.884491 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.884449 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-credential-socket\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.884656 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.884545 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-data\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.884656 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.884580 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istiod-ca-cert\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.884769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.884659 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-token\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.884769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.884696 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzh9h\" (UniqueName: \"kubernetes.io/projected/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-kube-api-access-mzh9h\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.884902 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.884775 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-podinfo\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.884902 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.884892 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-workload-certs\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.885014 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.884988 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-workload-socket\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.885067 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.885056 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-envoy\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.885758 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.885733 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:37:24.885862 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.885784 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" podUID="dfcede60-ff17-4d6a-b3ab-65dea8834b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 20:37:24.952124 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.952074 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v"] Nov 02 20:37:24.987074 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.987039 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-credential-socket\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.987340 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.987321 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-data\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.987463 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.987449 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istiod-ca-cert\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.987615 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.987603 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-token\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.987726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.987704 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzh9h\" (UniqueName: \"kubernetes.io/projected/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-kube-api-access-mzh9h\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.987911 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.987896 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-podinfo\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.988130 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.988115 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-workload-certs\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.988280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.988267 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-workload-socket\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.988406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.988394 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-envoy\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.989345 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.989323 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-credential-socket\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.989892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.989872 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-data\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.990608 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.990589 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istiod-ca-cert\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.993879 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.993846 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-workload-certs\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:24.996562 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:24.996528 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-workload-socket\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:25.007559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.007535 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-envoy\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:25.013627 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.013593 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-podinfo\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:25.056996 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.056966 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-token\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:25.062798 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.062769 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzh9h\" (UniqueName: \"kubernetes.io/projected/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-kube-api-access-mzh9h\") pod \"e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:25.275654 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.275561 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:25.288669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.288632 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:25.705388 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.705358 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" event={"ID":"a173d1e4-942e-4773-b9a4-ae854a61072d","Type":"ContainerStarted","Data":"c6e9487e761ad1c9d0ed5254307cb496e8c75c5e56ca4e32ccfc06e8e35b38b2"} Nov 02 20:37:25.705714 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.705420 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" event={"ID":"a173d1e4-942e-4773-b9a4-ae854a61072d","Type":"ContainerStarted","Data":"de445795353cc4042293681b672b1a2b14f1da666595e6a5d8c2875881a6ce16"} Nov 02 20:37:25.707183 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.707158 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" event={"ID":"2910c32a-6f07-47fd-8b35-bade9e68d34d","Type":"ContainerStarted","Data":"922776ecef14e0cdd0fde89871398cc3f6e30abf171506e33efaf6790c64c6ef"} Nov 02 20:37:25.758773 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.758738 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:37:25.758873 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.758795 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" podUID="dfcede60-ff17-4d6a-b3ab-65dea8834b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 20:37:25.789328 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.789302 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9"] Nov 02 20:37:25.790672 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:25.790643 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5847fbc4_1639_4ed1_ae14_182e5ab8b50f.slice/crio-f1fec747db8c4b2a65b32422da5bd93cfd4894b0f87e1eeb79524e6b11df1f2b WatchSource:0}: Error finding container f1fec747db8c4b2a65b32422da5bd93cfd4894b0f87e1eeb79524e6b11df1f2b: Status 404 returned error can't find the container with id f1fec747db8c4b2a65b32422da5bd93cfd4894b0f87e1eeb79524e6b11df1f2b Nov 02 20:37:25.793561 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.793527 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:25.793652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.793606 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:25.793711 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.793652 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:25.959182 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.957676 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" podStartSLOduration=2.957658071 podStartE2EDuration="2.957658071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:25.955676061 +0000 UTC m=+1374.245332592" watchObservedRunningTime="2026-11-02 20:37:25.957658071 +0000 UTC m=+1374.247314600" Nov 02 20:37:25.959182 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.958594 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" podStartSLOduration=1.958583481 podStartE2EDuration="1.958583481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:25.84513727 +0000 UTC m=+1374.134793799" watchObservedRunningTime="2026-11-02 20:37:25.958583481 +0000 UTC m=+1374.248240010" Nov 02 20:37:25.987384 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:25.987353 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-cd6d2c" Nov 02 20:37:26.062675 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.062644 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk"] Nov 02 20:37:26.067692 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.067669 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6539a1ee-lcsxk"] Nov 02 20:37:26.107286 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.107260 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a44d513-0188-4dc4-a6ef-e4214d30d9ef-kube-api-access-fk26b\" (UniqueName: \"kubernetes.io/projected/7a44d513-0188-4dc4-a6ef-e4214d30d9ef-kube-api-access-fk26b\") pod \"7a44d513-0188-4dc4-a6ef-e4214d30d9ef\" (UID: \"7a44d513-0188-4dc4-a6ef-e4214d30d9ef\") " Nov 02 20:37:26.109559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.109528 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a44d513-0188-4dc4-a6ef-e4214d30d9ef-kube-api-access-fk26b" pod "7a44d513-0188-4dc4-a6ef-e4214d30d9ef" (UID: "7a44d513-0188-4dc4-a6ef-e4214d30d9ef"). InnerVolumeSpecName "kube-api-access-fk26b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:26.209467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.209381 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk26b\" (UniqueName: \"kubernetes.io/projected/7a44d513-0188-4dc4-a6ef-e4214d30d9ef-kube-api-access-fk26b\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:26.717804 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.717765 4971 generic.go:356] "Generic (PLEG): container finished" podID="2910c32a-6f07-47fd-8b35-bade9e68d34d" containerID="922776ecef14e0cdd0fde89871398cc3f6e30abf171506e33efaf6790c64c6ef" exitCode=0 Nov 02 20:37:26.718289 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.717860 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" event={"ID":"2910c32a-6f07-47fd-8b35-bade9e68d34d","Type":"ContainerDied","Data":"922776ecef14e0cdd0fde89871398cc3f6e30abf171506e33efaf6790c64c6ef"} Nov 02 20:37:26.719402 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.719377 4971 generic.go:356] "Generic (PLEG): container finished" podID="7a44d513-0188-4dc4-a6ef-e4214d30d9ef" containerID="669bdb6614b05221584ea9d2241ee32b0d056f8be2ac997cd818371e48ba2924" exitCode=137 Nov 02 20:37:26.719518 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.719416 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7322-cd6d2c" Nov 02 20:37:26.719518 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.719470 4971 scope.go:122] "RemoveContainer" containerID="669bdb6614b05221584ea9d2241ee32b0d056f8be2ac997cd818371e48ba2924" Nov 02 20:37:26.721561 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.721534 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" event={"ID":"5847fbc4-1639-4ed1-ae14-182e5ab8b50f","Type":"ContainerStarted","Data":"f7b0bbcf8e2d925808102f49126643080e8a7e80ed70ad3c4cd2b23c1a27a3dc"} Nov 02 20:37:26.721668 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.721595 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" event={"ID":"5847fbc4-1639-4ed1-ae14-182e5ab8b50f","Type":"ContainerStarted","Data":"f1fec747db8c4b2a65b32422da5bd93cfd4894b0f87e1eeb79524e6b11df1f2b"} Nov 02 20:37:26.750065 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.750035 4971 generic.go:356] "Generic (PLEG): container finished" podID="a173d1e4-942e-4773-b9a4-ae854a61072d" containerID="c6e9487e761ad1c9d0ed5254307cb496e8c75c5e56ca4e32ccfc06e8e35b38b2" exitCode=0 Nov 02 20:37:26.750231 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.750122 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" event={"ID":"a173d1e4-942e-4773-b9a4-ae854a61072d","Type":"ContainerDied","Data":"c6e9487e761ad1c9d0ed5254307cb496e8c75c5e56ca4e32ccfc06e8e35b38b2"} Nov 02 20:37:26.754582 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.754562 4971 scope.go:122] "RemoveContainer" containerID="669bdb6614b05221584ea9d2241ee32b0d056f8be2ac997cd818371e48ba2924" Nov 02 20:37:26.754958 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:26.754931 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"669bdb6614b05221584ea9d2241ee32b0d056f8be2ac997cd818371e48ba2924\": container with ID starting with 669bdb6614b05221584ea9d2241ee32b0d056f8be2ac997cd818371e48ba2924 not found: ID does not exist" containerID="669bdb6614b05221584ea9d2241ee32b0d056f8be2ac997cd818371e48ba2924" Nov 02 20:37:26.755051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.754972 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"669bdb6614b05221584ea9d2241ee32b0d056f8be2ac997cd818371e48ba2924"} err="failed to get container status \"669bdb6614b05221584ea9d2241ee32b0d056f8be2ac997cd818371e48ba2924\": rpc error: code = NotFound desc = could not find container \"669bdb6614b05221584ea9d2241ee32b0d056f8be2ac997cd818371e48ba2924\": container with ID starting with 669bdb6614b05221584ea9d2241ee32b0d056f8be2ac997cd818371e48ba2924 not found: ID does not exist" Nov 02 20:37:26.758344 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.758315 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:37:26.758436 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:26.758357 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" podUID="dfcede60-ff17-4d6a-b3ab-65dea8834b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 20:37:26.894616 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:26.894578 4971 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-cd6d2c\" is forbidden: User \"system:node:ip-10-0-140-120.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-120.us-west-2.compute.internal' and this object" podUID="7a44d513-0188-4dc4-a6ef-e4214d30d9ef" pod="opendatahub/test-curl-7322-cd6d2c" Nov 02 20:37:27.011972 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:27.011884 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" podStartSLOduration=3.011870696 podStartE2EDuration="3.011870696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:27.008748725 +0000 UTC m=+1375.298405252" watchObservedRunningTime="2026-11-02 20:37:27.011870696 +0000 UTC m=+1375.301527223" Nov 02 20:37:27.289711 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:27.289625 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:27.295124 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:27.295099 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:27.347506 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:27.347286 4971 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-cd6d2c\" is forbidden: User \"system:node:ip-10-0-140-120.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-120.us-west-2.compute.internal' and this object" podUID="7a44d513-0188-4dc4-a6ef-e4214d30d9ef" pod="opendatahub/test-curl-7322-cd6d2c" Nov 02 20:37:27.757970 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:27.757936 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:37:27.758450 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:27.757994 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" podUID="dfcede60-ff17-4d6a-b3ab-65dea8834b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 20:37:27.760819 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:27.760486 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:27.761217 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:27.761198 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:37:27.809618 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:27.809561 4971 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7322-cd6d2c\" is forbidden: User \"system:node:ip-10-0-140-120.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-120.us-west-2.compute.internal' and this object" podUID="7a44d513-0188-4dc4-a6ef-e4214d30d9ef" pod="opendatahub/test-curl-7322-cd6d2c" Nov 02 20:37:27.872224 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:27.871538 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0576b7e9-c351-4260-86bf-10e795f98f46" path="/var/lib/kubelet/pods/0576b7e9-c351-4260-86bf-10e795f98f46/volumes" Nov 02 20:37:27.872224 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:27.871991 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a44d513-0188-4dc4-a6ef-e4214d30d9ef" path="/var/lib/kubelet/pods/7a44d513-0188-4dc4-a6ef-e4214d30d9ef/volumes" Nov 02 20:37:28.063777 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.063741 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" Nov 02 20:37:28.069374 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.069351 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" Nov 02 20:37:28.165133 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.165071 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2910c32a-6f07-47fd-8b35-bade9e68d34d-kube-api-access-vjpn5\" (UniqueName: \"kubernetes.io/projected/2910c32a-6f07-47fd-8b35-bade9e68d34d-kube-api-access-vjpn5\") pod \"2910c32a-6f07-47fd-8b35-bade9e68d34d\" (UID: \"2910c32a-6f07-47fd-8b35-bade9e68d34d\") " Nov 02 20:37:28.165315 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.165155 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2910c32a-6f07-47fd-8b35-bade9e68d34d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2910c32a-6f07-47fd-8b35-bade9e68d34d-maas-api-openshift-ca\") pod \"2910c32a-6f07-47fd-8b35-bade9e68d34d\" (UID: \"2910c32a-6f07-47fd-8b35-bade9e68d34d\") " Nov 02 20:37:28.165315 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.165198 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a173d1e4-942e-4773-b9a4-ae854a61072d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a173d1e4-942e-4773-b9a4-ae854a61072d-maas-api-xks-ca\") pod \"a173d1e4-942e-4773-b9a4-ae854a61072d\" (UID: \"a173d1e4-942e-4773-b9a4-ae854a61072d\") " Nov 02 20:37:28.165315 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.165229 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a173d1e4-942e-4773-b9a4-ae854a61072d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a173d1e4-942e-4773-b9a4-ae854a61072d-maas-api-openshift-ca\") pod \"a173d1e4-942e-4773-b9a4-ae854a61072d\" (UID: \"a173d1e4-942e-4773-b9a4-ae854a61072d\") " Nov 02 20:37:28.165471 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.165459 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a173d1e4-942e-4773-b9a4-ae854a61072d-kube-api-access-q2rrl\" (UniqueName: \"kubernetes.io/projected/a173d1e4-942e-4773-b9a4-ae854a61072d-kube-api-access-q2rrl\") pod \"a173d1e4-942e-4773-b9a4-ae854a61072d\" (UID: \"a173d1e4-942e-4773-b9a4-ae854a61072d\") " Nov 02 20:37:28.165524 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.165513 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2910c32a-6f07-47fd-8b35-bade9e68d34d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2910c32a-6f07-47fd-8b35-bade9e68d34d-maas-api-xks-ca\") pod \"2910c32a-6f07-47fd-8b35-bade9e68d34d\" (UID: \"2910c32a-6f07-47fd-8b35-bade9e68d34d\") " Nov 02 20:37:28.165583 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.165556 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2910c32a-6f07-47fd-8b35-bade9e68d34d-maas-api-openshift-ca" pod "2910c32a-6f07-47fd-8b35-bade9e68d34d" (UID: "2910c32a-6f07-47fd-8b35-bade9e68d34d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:28.165632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.165578 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a173d1e4-942e-4773-b9a4-ae854a61072d-maas-api-openshift-ca" pod "a173d1e4-942e-4773-b9a4-ae854a61072d" (UID: "a173d1e4-942e-4773-b9a4-ae854a61072d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:28.166041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.166022 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2910c32a-6f07-47fd-8b35-bade9e68d34d-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:28.166163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.166046 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a173d1e4-942e-4773-b9a4-ae854a61072d-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:28.167740 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.167710 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a173d1e4-942e-4773-b9a4-ae854a61072d-kube-api-access-q2rrl" pod "a173d1e4-942e-4773-b9a4-ae854a61072d" (UID: "a173d1e4-942e-4773-b9a4-ae854a61072d"). InnerVolumeSpecName "kube-api-access-q2rrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:28.167877 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.167855 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2910c32a-6f07-47fd-8b35-bade9e68d34d-kube-api-access-vjpn5" pod "2910c32a-6f07-47fd-8b35-bade9e68d34d" (UID: "2910c32a-6f07-47fd-8b35-bade9e68d34d"). InnerVolumeSpecName "kube-api-access-vjpn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:28.168033 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.168007 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a173d1e4-942e-4773-b9a4-ae854a61072d-maas-api-xks-ca" pod "a173d1e4-942e-4773-b9a4-ae854a61072d" (UID: "a173d1e4-942e-4773-b9a4-ae854a61072d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:28.168387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.168365 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2910c32a-6f07-47fd-8b35-bade9e68d34d-maas-api-xks-ca" pod "2910c32a-6f07-47fd-8b35-bade9e68d34d" (UID: "2910c32a-6f07-47fd-8b35-bade9e68d34d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:28.267310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.267238 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjpn5\" (UniqueName: \"kubernetes.io/projected/2910c32a-6f07-47fd-8b35-bade9e68d34d-kube-api-access-vjpn5\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:28.267310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.267264 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a173d1e4-942e-4773-b9a4-ae854a61072d-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:28.267310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.267275 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2rrl\" (UniqueName: \"kubernetes.io/projected/a173d1e4-942e-4773-b9a4-ae854a61072d-kube-api-access-q2rrl\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:28.267310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.267284 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2910c32a-6f07-47fd-8b35-bade9e68d34d-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:28.758411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.758381 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:37:28.758838 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.758434 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" podUID="dfcede60-ff17-4d6a-b3ab-65dea8834b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 02 20:37:28.771878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.771855 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" Nov 02 20:37:28.772032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.771896 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v" event={"ID":"a173d1e4-942e-4773-b9a4-ae854a61072d","Type":"ContainerDied","Data":"de445795353cc4042293681b672b1a2b14f1da666595e6a5d8c2875881a6ce16"} Nov 02 20:37:28.772032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.771947 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de445795353cc4042293681b672b1a2b14f1da666595e6a5d8c2875881a6ce16" Nov 02 20:37:28.773767 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.773737 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" Nov 02 20:37:28.773920 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.773767 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh" event={"ID":"2910c32a-6f07-47fd-8b35-bade9e68d34d","Type":"ContainerDied","Data":"f5f220a9d4bc3c9cf2fa9553369b2f3536a66a89183ca9658c176d63dd5b3b36"} Nov 02 20:37:28.773920 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:28.773833 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5f220a9d4bc3c9cf2fa9553369b2f3536a66a89183ca9658c176d63dd5b3b36" Nov 02 20:37:29.019159 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.010885 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:29.030102 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.020698 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj"] Nov 02 20:37:29.030102 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.021863 4971 cpu_manager.go:414] "containerMap: removing container" podUID="2910c32a-6f07-47fd-8b35-bade9e68d34d" containerName="revoke-keys" Nov 02 20:37:29.030102 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.021885 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2910c32a-6f07-47fd-8b35-bade9e68d34d" containerName="revoke-keys" Nov 02 20:37:29.030102 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.021946 4971 cpu_manager.go:414] "containerMap: removing container" podUID="a173d1e4-942e-4773-b9a4-ae854a61072d" containerName="revoke-keys" Nov 02 20:37:29.030102 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.021955 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a173d1e4-942e-4773-b9a4-ae854a61072d" containerName="revoke-keys" Nov 02 20:37:29.030102 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.022140 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="a173d1e4-942e-4773-b9a4-ae854a61072d" containerName="revoke-keys" Nov 02 20:37:29.030102 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.022166 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="2910c32a-6f07-47fd-8b35-bade9e68d34d" containerName="revoke-keys" Nov 02 20:37:29.066105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.053421 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" Nov 02 20:37:29.083006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.082981 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj"] Nov 02 20:37:29.110244 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.108049 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:37:29.110244 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.108163 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:37:29.156860 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.156563 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8"] Nov 02 20:37:29.164121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.164066 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a26d24-sbrp8"] Nov 02 20:37:29.202949 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.202901 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4d09ac-83dd-4b68-8784-6b5b959498a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2c2eff6-t6zbj\" (UID: \"db4d09ac-83dd-4b68-8784-6b5b959498a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" Nov 02 20:37:29.203341 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.203311 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4d09ac-83dd-4b68-8784-6b5b959498a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2c2eff6-t6zbj\" (UID: \"db4d09ac-83dd-4b68-8784-6b5b959498a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" Nov 02 20:37:29.203741 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.203719 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vc6r\" (UniqueName: \"kubernetes.io/projected/db4d09ac-83dd-4b68-8784-6b5b959498a2-kube-api-access-2vc6r\") pod \"maas-api-revoke-sub-d2c2eff6-t6zbj\" (UID: \"db4d09ac-83dd-4b68-8784-6b5b959498a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" Nov 02 20:37:29.256868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.256824 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:29.305282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.305205 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vc6r\" (UniqueName: \"kubernetes.io/projected/db4d09ac-83dd-4b68-8784-6b5b959498a2-kube-api-access-2vc6r\") pod \"maas-api-revoke-sub-d2c2eff6-t6zbj\" (UID: \"db4d09ac-83dd-4b68-8784-6b5b959498a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" Nov 02 20:37:29.305282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.305251 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4d09ac-83dd-4b68-8784-6b5b959498a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2c2eff6-t6zbj\" (UID: \"db4d09ac-83dd-4b68-8784-6b5b959498a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" Nov 02 20:37:29.305485 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.305338 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4d09ac-83dd-4b68-8784-6b5b959498a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2c2eff6-t6zbj\" (UID: \"db4d09ac-83dd-4b68-8784-6b5b959498a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" Nov 02 20:37:29.306029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.306000 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4d09ac-83dd-4b68-8784-6b5b959498a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2c2eff6-t6zbj\" (UID: \"db4d09ac-83dd-4b68-8784-6b5b959498a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" Nov 02 20:37:29.307844 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.307824 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4d09ac-83dd-4b68-8784-6b5b959498a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2c2eff6-t6zbj\" (UID: \"db4d09ac-83dd-4b68-8784-6b5b959498a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" Nov 02 20:37:29.508400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.508359 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:29.523123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.523095 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vc6r\" (UniqueName: \"kubernetes.io/projected/db4d09ac-83dd-4b68-8784-6b5b959498a2-kube-api-access-2vc6r\") pod \"maas-api-revoke-sub-d2c2eff6-t6zbj\" (UID: \"db4d09ac-83dd-4b68-8784-6b5b959498a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" Nov 02 20:37:29.751249 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.751216 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" Nov 02 20:37:29.803370 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.803333 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:29.963109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:29.960809 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f630eb29-efe9-490c-b8e5-8ecc7de298cc" path="/var/lib/kubelet/pods/f630eb29-efe9-490c-b8e5-8ecc7de298cc/volumes" Nov 02 20:37:30.627132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:30.606453 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:30.678387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:30.666031 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:30.679769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:30.679738 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:30.723389 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:30.721304 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj"] Nov 02 20:37:30.728951 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:30.728920 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:37:30.914305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:30.909161 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:30.914305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:30.909429 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 20:37:30.951135 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:30.917144 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" event={"ID":"db4d09ac-83dd-4b68-8784-6b5b959498a2","Type":"ContainerStarted","Data":"07a24925a94e0ff860d78907d12d7b58cb25d1750f2abb63acf1d8fe179f88b4"} Nov 02 20:37:30.951135 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:30.917221 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:30.951135 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:30.927669 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:30.983013 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:30.968160 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:31.027368 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:30.983695 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 20:37:31.132162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.115205 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:31.401758 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.401273 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:31.486109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.466174 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7"] Nov 02 20:37:31.599191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.596446 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.646005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.642446 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:31.646005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.645260 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:31.658379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.657828 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-6c66f658-gw-openshift-default-dockercfg-vrrfr\"" Nov 02 20:37:31.812942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.658751 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7"] Nov 02 20:37:31.813382 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.692630 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:31.813588 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.811439 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-credential-socket\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.813746 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.813727 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-token\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.813969 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.813950 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-workload-certs\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.814148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.814130 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fee3004b-ec0b-4c68-bcab-23b727eb7838-istiod-ca-cert\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.821610 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.821574 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-workload-socket\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.821781 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.821759 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-data\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.821921 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.821902 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td6r7\" (UniqueName: \"kubernetes.io/projected/fee3004b-ec0b-4c68-bcab-23b727eb7838-kube-api-access-td6r7\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.822062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.822043 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-envoy\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.827163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.822411 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-podinfo\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.916173 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.911371 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:31.916173 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.911544 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:31.955145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.932216 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:31.955145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.932413 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 20:37:31.955145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.933381 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-podinfo\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.955145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.933516 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-credential-socket\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.955145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.933549 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-token\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.955145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.933636 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-workload-certs\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.955145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.933698 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fee3004b-ec0b-4c68-bcab-23b727eb7838-istiod-ca-cert\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.955145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.933738 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-workload-socket\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.955145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.933771 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-data\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.955145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.933812 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td6r7\" (UniqueName: \"kubernetes.io/projected/fee3004b-ec0b-4c68-bcab-23b727eb7838-kube-api-access-td6r7\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.955145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.933860 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-envoy\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.955145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.937484 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-credential-socket\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.955145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.937672 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-data\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.955145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.937954 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-workload-certs\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.955145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.938490 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fee3004b-ec0b-4c68-bcab-23b727eb7838-istiod-ca-cert\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.964202 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.959486 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-workload-socket\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:31.997107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.970471 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:31.997107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:31.995311 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:32.068420 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.068129 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-podinfo\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:32.083122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.077365 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-envoy\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:32.120493 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.120337 4971 scope.go:122] "RemoveContainer" containerID="5d4bea43888a31369f5bc36cdcab5e378f1cf25d9a4f16f28416e196ac6bb540" Nov 02 20:37:32.125112 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.121815 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td6r7\" (UniqueName: \"kubernetes.io/projected/fee3004b-ec0b-4c68-bcab-23b727eb7838-kube-api-access-td6r7\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:32.131118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.125368 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:37:32.131118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.125422 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" podUID="ea994d9c-d955-4efa-9eea-9225a6fee64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:32.135892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.135629 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:32.150744 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.150702 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-token\") pod \"e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:32.159201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.159169 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 02 20:37:32.208145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.207889 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:32.360109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.358776 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:37:32.360109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.358835 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:37:32.360109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.358867 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:37:32.360109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.358897 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:37:32.360109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.358936 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:37:32.360109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.358967 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:37:32.360109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.358983 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" event={"ID":"db4d09ac-83dd-4b68-8784-6b5b959498a2","Type":"ContainerStarted","Data":"e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b"} Nov 02 20:37:32.419814 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.418433 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-6c66f658-gw-openshift-default-dockercfg-vrrfr\"" Nov 02 20:37:32.419814 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.418977 4971 scope.go:122] "RemoveContainer" containerID="b5eed181be6d5ed7661e4fbf5c46fdf93b1ad20b3232c0a0c6d2659c16e67bb1" Nov 02 20:37:32.419814 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.419152 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:32.472984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.472941 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:32.533332 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.533270 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" podStartSLOduration=4.533248009 podStartE2EDuration="4.533248009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:32.502495733 +0000 UTC m=+1380.792152292" watchObservedRunningTime="2026-11-02 20:37:32.533248009 +0000 UTC m=+1380.822904538" Nov 02 20:37:32.884200 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.651712 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:32.884671 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.884645 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:37:32.910206 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.651851 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 02 20:37:32.910206 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.890681 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:37:32.910206 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:32.697433 4971 scope.go:122] "RemoveContainer" containerID="36fbd1ce92fcd73a96145a1df36c32a0e3c77d90715248bab9d0479a1eae8256" Nov 02 20:37:33.403988 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.403933 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:33.415125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.407624 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:33.435108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.416246 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:33.439978 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.439935 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:33.863698 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.452876 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 02 20:37:33.990996 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.466234 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:33.991354 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.466538 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:37:33.991515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.723507 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:33.991704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.724017 4971 scope.go:122] "RemoveContainer" containerID="90d8c26a6aadbeb14c091babdfeb72a9114e14cdb0e66f1e134746625b83b8b2" Nov 02 20:37:33.991919 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.767706 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 02 20:37:33.992112 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.767823 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 20:37:33.992313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.805239 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 20:37:33.992511 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.809257 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:37:33.996106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.992643 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" podUID="413b0cf5-5c2e-4983-8437-7f9d2e899405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:33.996302 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.872324 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:37:33.996444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.996420 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" podUID="191d909c-208a-43e3-91ad-9d0f979b1cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:34.021110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:33.863310 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 20:37:34.266177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:34.266115 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hc8gn" podUID="f0316745-a138-4b26-9f2a-a9f5ffd9d97c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:34.359117 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:34.350398 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:34.429605 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:34.398496 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:34.466111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:34.458218 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:34.466111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:34.458345 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:34.466111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:34.459138 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 20:37:34.466111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:34.459434 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded" Nov 02 20:37:34.921926 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:34.921864 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:35.065167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:34.927360 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:35.100112 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:34.947879 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:35.100341 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:34.993188 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:35.100694 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:34.993234 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 02 20:37:35.101118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:34.993255 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 20:37:35.101487 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:34.993275 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:35.105508 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:35.039725 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:35.109107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:35.072299 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Nov 02 20:37:35.472232 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:35.458813 4971 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.393s" Nov 02 20:37:35.472899 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:35.472875 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:37:35.497116 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:35.473064 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:37:35.498347 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:35.466174 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 02 20:37:35.498849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:35.468634 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 02 20:37:35.500796 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:35.500761 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:35.531992 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:35.531625 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:35.563635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:35.533388 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:35.815111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:35.809070 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:35.815111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:35.809219 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:35.815111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:35.809294 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:35.933160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:35.923185 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:36.155139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.128160 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:36.155139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.131228 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:36.155139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.146157 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 20:37:36.155139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.146424 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 02 20:37:36.155139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.147268 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 02 20:37:36.188355 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.187229 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:36.215110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.199419 4971 generic.go:356] "Generic (PLEG): container finished" podID="db4d09ac-83dd-4b68-8784-6b5b959498a2" containerID="e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b" exitCode=0 Nov 02 20:37:36.330387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.330351 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm"] Nov 02 20:37:36.330639 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.330624 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcfbe15b-kcjjm"] Nov 02 20:37:36.330755 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.330737 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" event={"ID":"db4d09ac-83dd-4b68-8784-6b5b959498a2","Type":"ContainerDied","Data":"e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b"} Nov 02 20:37:36.509531 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:36.489734 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:37:36.700651 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.699916 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:36.701526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.700984 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:36.712898 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.710469 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:37:36.715333 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.715311 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:37:36.717176 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.716620 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:36.729131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.728936 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:37:36.729131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.728998 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:37:36.729131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.729036 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:37:36.731222 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.730928 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:37:36.731222 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.730980 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:37:36.741835 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.741667 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:37:36.750103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.746430 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7"] Nov 02 20:37:36.761439 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.761415 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:37:36.762369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.762346 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:37:36.797160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:36.797138 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:37:36.848542 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:36.848499 4971 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-conmon-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:37:37.209966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:37.209934 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" event={"ID":"fee3004b-ec0b-4c68-bcab-23b727eb7838","Type":"ContainerStarted","Data":"8ba9cee98bed6ceb1b84bc49b0dd24f01f1ea01fb5cb42b0135ab93aebc54641"} Nov 02 20:37:37.210180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:37.209987 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" event={"ID":"fee3004b-ec0b-4c68-bcab-23b727eb7838","Type":"ContainerStarted","Data":"9b80c6ac7114933811187734df5098d2e900ba35beac9ff74750f90c8d02461a"} Nov 02 20:37:37.309589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:37.309554 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:37.389978 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:37.389952 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" Nov 02 20:37:37.421247 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:37.421219 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:37.426476 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:37.426450 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db4d09ac-83dd-4b68-8784-6b5b959498a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4d09ac-83dd-4b68-8784-6b5b959498a2-maas-api-xks-ca\") pod \"db4d09ac-83dd-4b68-8784-6b5b959498a2\" (UID: \"db4d09ac-83dd-4b68-8784-6b5b959498a2\") " Nov 02 20:37:37.426673 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:37.426551 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4d09ac-83dd-4b68-8784-6b5b959498a2-kube-api-access-2vc6r\" (UniqueName: \"kubernetes.io/projected/db4d09ac-83dd-4b68-8784-6b5b959498a2-kube-api-access-2vc6r\") pod \"db4d09ac-83dd-4b68-8784-6b5b959498a2\" (UID: \"db4d09ac-83dd-4b68-8784-6b5b959498a2\") " Nov 02 20:37:37.426770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:37.426748 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:37.426838 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:37.426789 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4d09ac-83dd-4b68-8784-6b5b959498a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4d09ac-83dd-4b68-8784-6b5b959498a2-maas-api-openshift-ca\") pod \"db4d09ac-83dd-4b68-8784-6b5b959498a2\" (UID: \"db4d09ac-83dd-4b68-8784-6b5b959498a2\") " Nov 02 20:37:37.427232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:37.427203 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4d09ac-83dd-4b68-8784-6b5b959498a2-maas-api-openshift-ca" pod "db4d09ac-83dd-4b68-8784-6b5b959498a2" (UID: "db4d09ac-83dd-4b68-8784-6b5b959498a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:37.427427 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:37.427411 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4d09ac-83dd-4b68-8784-6b5b959498a2-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:37.428720 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:37.428694 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db4d09ac-83dd-4b68-8784-6b5b959498a2-maas-api-xks-ca" pod "db4d09ac-83dd-4b68-8784-6b5b959498a2" (UID: "db4d09ac-83dd-4b68-8784-6b5b959498a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:37.428873 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:37.428855 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4d09ac-83dd-4b68-8784-6b5b959498a2-kube-api-access-2vc6r" pod "db4d09ac-83dd-4b68-8784-6b5b959498a2" (UID: "db4d09ac-83dd-4b68-8784-6b5b959498a2"). InnerVolumeSpecName "kube-api-access-2vc6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:37.529250 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:37.529179 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vc6r\" (UniqueName: \"kubernetes.io/projected/db4d09ac-83dd-4b68-8784-6b5b959498a2-kube-api-access-2vc6r\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:37.529250 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:37.529206 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4d09ac-83dd-4b68-8784-6b5b959498a2-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:37.870861 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:37.870801 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c659ab03-c476-47a4-91c9-0457bb538cd7" path="/var/lib/kubelet/pods/c659ab03-c476-47a4-91c9-0457bb538cd7/volumes" Nov 02 20:37:38.049464 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:38.049413 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" podStartSLOduration=7.049398906 podStartE2EDuration="7.049398906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:38.044598076 +0000 UTC m=+1386.334254605" watchObservedRunningTime="2026-11-02 20:37:38.049398906 +0000 UTC m=+1386.339055434" Nov 02 20:37:38.221049 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:38.220977 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" Nov 02 20:37:38.221049 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:38.221016 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj" event={"ID":"db4d09ac-83dd-4b68-8784-6b5b959498a2","Type":"ContainerDied","Data":"07a24925a94e0ff860d78907d12d7b58cb25d1750f2abb63acf1d8fe179f88b4"} Nov 02 20:37:38.221303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:38.221061 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07a24925a94e0ff860d78907d12d7b58cb25d1750f2abb63acf1d8fe179f88b4" Nov 02 20:37:38.221448 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:38.221426 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:38.222324 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:38.222304 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:37:38.961054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:38.961019 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:39.241616 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.241494 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:39.300150 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:39.300064 4971 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-conmon-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:37:39.451701 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.450591 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr"] Nov 02 20:37:39.451701 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.451709 4971 cpu_manager.go:414] "containerMap: removing container" podUID="db4d09ac-83dd-4b68-8784-6b5b959498a2" containerName="revoke-keys" Nov 02 20:37:39.452076 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.451727 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4d09ac-83dd-4b68-8784-6b5b959498a2" containerName="revoke-keys" Nov 02 20:37:39.452076 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.451829 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4d09ac-83dd-4b68-8784-6b5b959498a2" containerName="revoke-keys" Nov 02 20:37:39.457686 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.457658 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" Nov 02 20:37:39.495646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.495542 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:39.508880 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.508846 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr"] Nov 02 20:37:39.509126 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.509110 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2b3e20d8\"" Nov 02 20:37:39.544352 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.544322 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr\" (UID: \"e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" Nov 02 20:37:39.544540 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.544520 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c76q7\" (UniqueName: \"kubernetes.io/projected/e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17-kube-api-access-c76q7\") pod \"payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr\" (UID: \"e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" Nov 02 20:37:39.574510 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.574477 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p"] Nov 02 20:37:39.579751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.579723 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" Nov 02 20:37:39.630509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.630475 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2b3e20d8-dockercfg-lgrqx\"" Nov 02 20:37:39.635236 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.635204 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p"] Nov 02 20:37:39.645488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.645459 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c76q7\" (UniqueName: \"kubernetes.io/projected/e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17-kube-api-access-c76q7\") pod \"payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr\" (UID: \"e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" Nov 02 20:37:39.645673 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.645511 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wz4q\" (UniqueName: \"kubernetes.io/projected/9b9eab18-7c27-4e6c-9bf2-bdfab47f0610-kube-api-access-8wz4q\") pod \"payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p\" (UID: \"9b9eab18-7c27-4e6c-9bf2-bdfab47f0610\") " pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" Nov 02 20:37:39.645766 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.645717 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr\" (UID: \"e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" Nov 02 20:37:39.645839 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.645776 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9eab18-7c27-4e6c-9bf2-bdfab47f0610-plugins-config-volume\") pod \"payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p\" (UID: \"9b9eab18-7c27-4e6c-9bf2-bdfab47f0610\") " pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" Nov 02 20:37:39.646715 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.646689 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr\" (UID: \"e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" Nov 02 20:37:39.756685 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.756374 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9eab18-7c27-4e6c-9bf2-bdfab47f0610-plugins-config-volume\") pod \"payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p\" (UID: \"9b9eab18-7c27-4e6c-9bf2-bdfab47f0610\") " pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" Nov 02 20:37:39.757163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.757123 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wz4q\" (UniqueName: \"kubernetes.io/projected/9b9eab18-7c27-4e6c-9bf2-bdfab47f0610-kube-api-access-8wz4q\") pod \"payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p\" (UID: \"9b9eab18-7c27-4e6c-9bf2-bdfab47f0610\") " pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" Nov 02 20:37:39.758611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.758588 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9eab18-7c27-4e6c-9bf2-bdfab47f0610-plugins-config-volume\") pod \"payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p\" (UID: \"9b9eab18-7c27-4e6c-9bf2-bdfab47f0610\") " pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" Nov 02 20:37:39.777784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.777715 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c76q7\" (UniqueName: \"kubernetes.io/projected/e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17-kube-api-access-c76q7\") pod \"payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr\" (UID: \"e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" Nov 02 20:37:39.791509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.790774 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" Nov 02 20:37:39.828702 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.821334 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wz4q\" (UniqueName: \"kubernetes.io/projected/9b9eab18-7c27-4e6c-9bf2-bdfab47f0610-kube-api-access-8wz4q\") pod \"payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p\" (UID: \"9b9eab18-7c27-4e6c-9bf2-bdfab47f0610\") " pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" Nov 02 20:37:39.908847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:39.905410 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" Nov 02 20:37:40.057534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:40.057448 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:40.220559 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:40.220527 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26c8fd8_69ac_4f08_8fea_6ad4bfb7dc17.slice/crio-0cb3cb81effcff437b36d1ac3daafda2fac9a9ba2b0af9af281f11dcd543c96f WatchSource:0}: Error finding container 0cb3cb81effcff437b36d1ac3daafda2fac9a9ba2b0af9af281f11dcd543c96f: Status 404 returned error can't find the container with id 0cb3cb81effcff437b36d1ac3daafda2fac9a9ba2b0af9af281f11dcd543c96f Nov 02 20:37:40.222835 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:40.222715 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr"] Nov 02 20:37:40.297067 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:40.297034 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" event={"ID":"e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17","Type":"ContainerStarted","Data":"0cb3cb81effcff437b36d1ac3daafda2fac9a9ba2b0af9af281f11dcd543c96f"} Nov 02 20:37:40.325097 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:40.325016 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9eab18_7c27_4e6c_9bf2_bdfab47f0610.slice/crio-e91f83e4fe9ecf4a3fa71593f394b9b954e508e162ca94aa9330ec039de82fea WatchSource:0}: Error finding container e91f83e4fe9ecf4a3fa71593f394b9b954e508e162ca94aa9330ec039de82fea: Status 404 returned error can't find the container with id e91f83e4fe9ecf4a3fa71593f394b9b954e508e162ca94aa9330ec039de82fea Nov 02 20:37:40.326134 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:40.326107 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p"] Nov 02 20:37:40.871671 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:40.871636 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:41.124621 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.124531 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:41.307904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.307817 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" event={"ID":"9b9eab18-7c27-4e6c-9bf2-bdfab47f0610","Type":"ContainerStarted","Data":"12d523fbc167030310ffcc141cd8b18045cd5e1c4a836452c4ec255184bf14a1"} Nov 02 20:37:41.307904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.307877 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" event={"ID":"9b9eab18-7c27-4e6c-9bf2-bdfab47f0610","Type":"ContainerStarted","Data":"e91f83e4fe9ecf4a3fa71593f394b9b954e508e162ca94aa9330ec039de82fea"} Nov 02 20:37:41.308178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.307906 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" Nov 02 20:37:41.309810 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.309780 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" event={"ID":"e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17","Type":"ContainerStarted","Data":"9837563d65069e057c5ddbd865c7342134a6b0b1be4f107b93c3233b683a5a8e"} Nov 02 20:37:41.309941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.309854 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" Nov 02 20:37:41.456023 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.455900 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" podStartSLOduration=2.455862109 podStartE2EDuration="2.455862109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:41.450476658 +0000 UTC m=+1389.740133186" watchObservedRunningTime="2026-11-02 20:37:41.455862109 +0000 UTC m=+1389.745518638" Nov 02 20:37:41.575672 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.575611 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" podStartSLOduration=2.5755958249999997 podStartE2EDuration="2.575595825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:41.570756918 +0000 UTC m=+1389.860413446" watchObservedRunningTime="2026-11-02 20:37:41.575595825 +0000 UTC m=+1389.865252353" Nov 02 20:37:41.609811 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.609783 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65"] Nov 02 20:37:41.615614 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.615597 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.665308 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.665282 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-574357e2-openshift-default-dockercfg-f88s7\"" Nov 02 20:37:41.684379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.684352 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65"] Nov 02 20:37:41.831169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.829991 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:41.835106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.834103 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-data\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.835106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.834243 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-envoy\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.835106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.834280 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istiod-ca-cert\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.835106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.834312 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-workload-certs\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.835106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.834375 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-token\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.835106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.834402 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-podinfo\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.835106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.834522 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8wbf\" (UniqueName: \"kubernetes.io/projected/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-kube-api-access-f8wbf\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.835106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.834553 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-workload-socket\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.835106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.834613 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-credential-socket\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.935588 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.935552 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-data\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.935753 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.935669 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-envoy\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.935753 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.935695 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istiod-ca-cert\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.935753 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.935716 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-workload-certs\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.935908 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.935783 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-token\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.935908 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.935811 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-podinfo\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.936024 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.935944 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8wbf\" (UniqueName: \"kubernetes.io/projected/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-kube-api-access-f8wbf\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.936024 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.935957 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-data\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.936024 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.935975 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-workload-socket\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.936210 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.936055 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-credential-socket\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.936210 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.936164 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-workload-certs\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.936459 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.936432 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istiod-ca-cert\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.936739 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.936507 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-workload-socket\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.936739 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.936626 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-credential-socket\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.938368 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.938348 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-envoy\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.938765 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.938746 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-podinfo\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.994100 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.994059 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-token\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:41.995640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:41.995616 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8wbf\" (UniqueName: \"kubernetes.io/projected/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-kube-api-access-f8wbf\") pod \"e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:42.081285 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:42.081210 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:42.232679 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:42.232266 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:42.661447 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:42.661414 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc221b7fd_3fe6_4ddc_b09e_38eb310e9d7a.slice/crio-96dcb7fc94e658c65f5fe915222b90d7b7b206288d9da6833a231c4e45c100cb WatchSource:0}: Error finding container 96dcb7fc94e658c65f5fe915222b90d7b7b206288d9da6833a231c4e45c100cb: Status 404 returned error can't find the container with id 96dcb7fc94e658c65f5fe915222b90d7b7b206288d9da6833a231c4e45c100cb Nov 02 20:37:42.664039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:42.664004 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:42.664169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:42.664074 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:42.664169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:42.664153 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:42.664833 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:42.664810 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65"] Nov 02 20:37:42.986142 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:42.985999 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:43.241711 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.241626 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:43.341238 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.341197 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" event={"ID":"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a","Type":"ContainerStarted","Data":"123686016c6f8ee17ad443eb47bce41508ba670fb3e6b680aa1926e8abf5fca7"} Nov 02 20:37:43.341391 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.341289 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" event={"ID":"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a","Type":"ContainerStarted","Data":"96dcb7fc94e658c65f5fe915222b90d7b7b206288d9da6833a231c4e45c100cb"} Nov 02 20:37:43.484616 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.484584 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:43.503605 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.503520 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" podStartSLOduration=2.503507077 podStartE2EDuration="2.503507077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:43.502021533 +0000 UTC m=+1391.791678061" watchObservedRunningTime="2026-11-02 20:37:43.503507077 +0000 UTC m=+1391.793163601" Nov 02 20:37:43.582734 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.582706 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd"] Nov 02 20:37:43.587506 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.587488 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd" Nov 02 20:37:43.640983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.639960 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:37:43.640983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.640139 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:37:43.649137 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.647958 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd"] Nov 02 20:37:43.663926 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.663886 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e623aad5-lwrqd\" (UID: \"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd" Nov 02 20:37:43.664141 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.663994 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmtlg\" (UniqueName: \"kubernetes.io/projected/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-kube-api-access-fmtlg\") pod \"maas-api-revoke-sub-e623aad5-lwrqd\" (UID: \"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd" Nov 02 20:37:43.664209 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.664157 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e623aad5-lwrqd\" (UID: \"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd" Nov 02 20:37:43.765454 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.765372 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e623aad5-lwrqd\" (UID: \"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd" Nov 02 20:37:43.765628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.765530 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e623aad5-lwrqd\" (UID: \"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd" Nov 02 20:37:43.765628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.765600 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmtlg\" (UniqueName: \"kubernetes.io/projected/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-kube-api-access-fmtlg\") pod \"maas-api-revoke-sub-e623aad5-lwrqd\" (UID: \"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd" Nov 02 20:37:43.766250 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.766229 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e623aad5-lwrqd\" (UID: \"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd" Nov 02 20:37:43.768043 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.768021 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e623aad5-lwrqd\" (UID: \"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd" Nov 02 20:37:43.823359 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.823334 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmtlg\" (UniqueName: \"kubernetes.io/projected/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-kube-api-access-fmtlg\") pod \"maas-api-revoke-sub-e623aad5-lwrqd\" (UID: \"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd" Nov 02 20:37:43.902867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:43.902843 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd" Nov 02 20:37:44.233259 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:44.233229 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:44.239547 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:44.239520 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:44.307222 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:44.307185 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd2a50d_64c2_4ace_bc50_d3ac3d8c70fe.slice/crio-43fcf8690d48bccbe616f020f18693fb8cd1ddb4346b2afa32349b28e083169f WatchSource:0}: Error finding container 43fcf8690d48bccbe616f020f18693fb8cd1ddb4346b2afa32349b28e083169f: Status 404 returned error can't find the container with id 43fcf8690d48bccbe616f020f18693fb8cd1ddb4346b2afa32349b28e083169f Nov 02 20:37:44.311167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:44.311142 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd"] Nov 02 20:37:44.351183 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:44.351157 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd" event={"ID":"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe","Type":"ContainerStarted","Data":"43fcf8690d48bccbe616f020f18693fb8cd1ddb4346b2afa32349b28e083169f"} Nov 02 20:37:44.351584 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:44.351561 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:44.352422 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:44.352405 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:37:44.415534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:44.415502 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:44.662996 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:44.662968 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:45.361698 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:45.361663 4971 generic.go:356] "Generic (PLEG): container finished" podID="2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe" containerID="170e1ecd6dd043d2228a3b0e16d730007122a58a714be11d0c276d0041f1153c" exitCode=0 Nov 02 20:37:45.362118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:45.361793 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd" event={"ID":"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe","Type":"ContainerDied","Data":"170e1ecd6dd043d2228a3b0e16d730007122a58a714be11d0c276d0041f1153c"} Nov 02 20:37:45.643525 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:45.643495 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:45.909924 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:45.909800 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:46.558478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.558454 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd" Nov 02 20:37:46.604280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.604252 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-maas-api-xks-ca\") pod \"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe\" (UID: \"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe\") " Nov 02 20:37:46.604412 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.604362 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-kube-api-access-fmtlg\" (UniqueName: \"kubernetes.io/projected/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-kube-api-access-fmtlg\") pod \"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe\" (UID: \"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe\") " Nov 02 20:37:46.604471 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.604460 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-maas-api-openshift-ca\") pod \"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe\" (UID: \"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe\") " Nov 02 20:37:46.604809 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.604786 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-maas-api-openshift-ca" pod "2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe" (UID: "2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:46.604918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.604860 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:46.606429 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.606407 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-maas-api-xks-ca" pod "2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe" (UID: "2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:46.606581 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.606565 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-kube-api-access-fmtlg" pod "2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe" (UID: "2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe"). InnerVolumeSpecName "kube-api-access-fmtlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:46.706564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.706491 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmtlg\" (UniqueName: \"kubernetes.io/projected/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-kube-api-access-fmtlg\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:46.706564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.706519 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:46.725573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.725019 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55"] Nov 02 20:37:46.729469 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.729447 4971 cpu_manager.go:414] "containerMap: removing container" podUID="2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe" containerName="revoke-keys" Nov 02 20:37:46.729469 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.729466 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe" containerName="revoke-keys" Nov 02 20:37:46.729662 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.729619 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe" containerName="revoke-keys" Nov 02 20:37:46.733619 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.733600 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" Nov 02 20:37:46.782441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.782413 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55"] Nov 02 20:37:46.807533 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.807500 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcxv2\" (UniqueName: \"kubernetes.io/projected/fd952597-9010-4a13-83b6-f178f7942885-kube-api-access-pcxv2\") pod \"maas-api-revoke-sub-64df16ca-8nl55\" (UID: \"fd952597-9010-4a13-83b6-f178f7942885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" Nov 02 20:37:46.807633 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.807562 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd952597-9010-4a13-83b6-f178f7942885-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64df16ca-8nl55\" (UID: \"fd952597-9010-4a13-83b6-f178f7942885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" Nov 02 20:37:46.807779 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.807755 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd952597-9010-4a13-83b6-f178f7942885-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64df16ca-8nl55\" (UID: \"fd952597-9010-4a13-83b6-f178f7942885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" Nov 02 20:37:46.908897 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.908869 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd952597-9010-4a13-83b6-f178f7942885-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64df16ca-8nl55\" (UID: \"fd952597-9010-4a13-83b6-f178f7942885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" Nov 02 20:37:46.909027 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.909011 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcxv2\" (UniqueName: \"kubernetes.io/projected/fd952597-9010-4a13-83b6-f178f7942885-kube-api-access-pcxv2\") pod \"maas-api-revoke-sub-64df16ca-8nl55\" (UID: \"fd952597-9010-4a13-83b6-f178f7942885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" Nov 02 20:37:46.909069 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.909046 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd952597-9010-4a13-83b6-f178f7942885-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64df16ca-8nl55\" (UID: \"fd952597-9010-4a13-83b6-f178f7942885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" Nov 02 20:37:46.909520 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.909502 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd952597-9010-4a13-83b6-f178f7942885-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64df16ca-8nl55\" (UID: \"fd952597-9010-4a13-83b6-f178f7942885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" Nov 02 20:37:46.911548 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.911519 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd952597-9010-4a13-83b6-f178f7942885-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64df16ca-8nl55\" (UID: \"fd952597-9010-4a13-83b6-f178f7942885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" Nov 02 20:37:46.966830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:46.966773 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcxv2\" (UniqueName: \"kubernetes.io/projected/fd952597-9010-4a13-83b6-f178f7942885-kube-api-access-pcxv2\") pod \"maas-api-revoke-sub-64df16ca-8nl55\" (UID: \"fd952597-9010-4a13-83b6-f178f7942885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" Nov 02 20:37:47.050345 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:47.050321 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" Nov 02 20:37:47.381277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:47.381251 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd" Nov 02 20:37:47.381424 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:47.381289 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd" event={"ID":"2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe","Type":"ContainerDied","Data":"43fcf8690d48bccbe616f020f18693fb8cd1ddb4346b2afa32349b28e083169f"} Nov 02 20:37:47.381424 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:47.381338 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43fcf8690d48bccbe616f020f18693fb8cd1ddb4346b2afa32349b28e083169f" Nov 02 20:37:47.422369 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:47.422343 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd952597_9010_4a13_83b6_f178f7942885.slice/crio-d3852c7ae83f72f7d178f20a6f421a7bd731dccb3d3a77e84ade0edbf4375205 WatchSource:0}: Error finding container d3852c7ae83f72f7d178f20a6f421a7bd731dccb3d3a77e84ade0edbf4375205: Status 404 returned error can't find the container with id d3852c7ae83f72f7d178f20a6f421a7bd731dccb3d3a77e84ade0edbf4375205 Nov 02 20:37:47.424063 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:47.424042 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55"] Nov 02 20:37:48.244340 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:48.244304 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:48.393507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:48.393472 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" event={"ID":"fd952597-9010-4a13-83b6-f178f7942885","Type":"ContainerStarted","Data":"5e02678173f5494faaceb6ebaa8d2e86c3b19ac58506f2267d84d36b35e0d5f2"} Nov 02 20:37:48.393676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:48.393530 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" event={"ID":"fd952597-9010-4a13-83b6-f178f7942885","Type":"ContainerStarted","Data":"d3852c7ae83f72f7d178f20a6f421a7bd731dccb3d3a77e84ade0edbf4375205"} Nov 02 20:37:49.097116 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:49.096214 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:49.354445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:49.354353 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:49.407813 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:49.407775 4971 generic.go:356] "Generic (PLEG): container finished" podID="fd952597-9010-4a13-83b6-f178f7942885" containerID="5e02678173f5494faaceb6ebaa8d2e86c3b19ac58506f2267d84d36b35e0d5f2" exitCode=0 Nov 02 20:37:49.407983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:49.407863 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" event={"ID":"fd952597-9010-4a13-83b6-f178f7942885","Type":"ContainerDied","Data":"5e02678173f5494faaceb6ebaa8d2e86c3b19ac58506f2267d84d36b35e0d5f2"} Nov 02 20:37:49.419850 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:49.419816 4971 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-conmon-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:37:49.747487 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:49.747463 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" Nov 02 20:37:49.891151 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:49.891119 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd952597-9010-4a13-83b6-f178f7942885-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd952597-9010-4a13-83b6-f178f7942885-maas-api-openshift-ca\") pod \"fd952597-9010-4a13-83b6-f178f7942885\" (UID: \"fd952597-9010-4a13-83b6-f178f7942885\") " Nov 02 20:37:49.891326 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:49.891191 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd952597-9010-4a13-83b6-f178f7942885-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd952597-9010-4a13-83b6-f178f7942885-maas-api-xks-ca\") pod \"fd952597-9010-4a13-83b6-f178f7942885\" (UID: \"fd952597-9010-4a13-83b6-f178f7942885\") " Nov 02 20:37:49.891400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:49.891378 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd952597-9010-4a13-83b6-f178f7942885-kube-api-access-pcxv2\" (UniqueName: \"kubernetes.io/projected/fd952597-9010-4a13-83b6-f178f7942885-kube-api-access-pcxv2\") pod \"fd952597-9010-4a13-83b6-f178f7942885\" (UID: \"fd952597-9010-4a13-83b6-f178f7942885\") " Nov 02 20:37:49.891542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:49.891520 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd952597-9010-4a13-83b6-f178f7942885-maas-api-openshift-ca" pod "fd952597-9010-4a13-83b6-f178f7942885" (UID: "fd952597-9010-4a13-83b6-f178f7942885"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:49.892267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:49.892240 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd952597-9010-4a13-83b6-f178f7942885-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:49.893691 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:49.893662 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd952597-9010-4a13-83b6-f178f7942885-maas-api-xks-ca" pod "fd952597-9010-4a13-83b6-f178f7942885" (UID: "fd952597-9010-4a13-83b6-f178f7942885"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:49.894175 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:49.894148 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd952597-9010-4a13-83b6-f178f7942885-kube-api-access-pcxv2" pod "fd952597-9010-4a13-83b6-f178f7942885" (UID: "fd952597-9010-4a13-83b6-f178f7942885"). InnerVolumeSpecName "kube-api-access-pcxv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:49.993600 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:49.993532 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd952597-9010-4a13-83b6-f178f7942885-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:49.993600 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:49.993566 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcxv2\" (UniqueName: \"kubernetes.io/projected/fd952597-9010-4a13-83b6-f178f7942885-kube-api-access-pcxv2\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:50.419508 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:50.419484 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" Nov 02 20:37:50.419898 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:50.419525 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55" event={"ID":"fd952597-9010-4a13-83b6-f178f7942885","Type":"ContainerDied","Data":"d3852c7ae83f72f7d178f20a6f421a7bd731dccb3d3a77e84ade0edbf4375205"} Nov 02 20:37:50.419898 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:50.419567 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3852c7ae83f72f7d178f20a6f421a7bd731dccb3d3a77e84ade0edbf4375205" Nov 02 20:37:50.469844 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:50.469808 4971 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-conmon-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:37:51.790304 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:51.790264 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:52.044290 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.044199 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:52.347422 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.347393 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" Nov 02 20:37:52.363474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.363447 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" Nov 02 20:37:52.846326 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.846277 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6"] Nov 02 20:37:52.846943 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.846927 4971 cpu_manager.go:414] "containerMap: removing container" podUID="fd952597-9010-4a13-83b6-f178f7942885" containerName="revoke-keys" Nov 02 20:37:52.846987 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.846944 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd952597-9010-4a13-83b6-f178f7942885" containerName="revoke-keys" Nov 02 20:37:52.847116 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.847107 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd952597-9010-4a13-83b6-f178f7942885" containerName="revoke-keys" Nov 02 20:37:52.851313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.851291 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:52.939861 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.939831 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6"] Nov 02 20:37:52.991558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.991290 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6vmw\" (UniqueName: \"kubernetes.io/projected/d0ed161a-7533-42b6-94c9-bfaacc784178-kube-api-access-b6vmw\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:52.991558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.991345 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:52.991558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.991395 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:52.991558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.991430 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:52.991558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.991461 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ed161a-7533-42b6-94c9-bfaacc784178-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:52.991960 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.991602 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:52.991960 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.991684 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:52.991960 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.991741 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:52.991960 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:52.991809 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.092689 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.092658 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6vmw\" (UniqueName: \"kubernetes.io/projected/d0ed161a-7533-42b6-94c9-bfaacc784178-kube-api-access-b6vmw\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.092878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.092695 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.092878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.092824 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.092878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.092869 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.093058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.092899 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ed161a-7533-42b6-94c9-bfaacc784178-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.093058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.093015 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.093204 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.093065 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.093204 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.093168 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.093306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.093207 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.093306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.093262 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.093306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.093275 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.093474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.093377 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.093580 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.093561 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.093700 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.093681 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ed161a-7533-42b6-94c9-bfaacc784178-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.095330 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.095312 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.095785 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.095769 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.135642 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.135586 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v"] Nov 02 20:37:53.140488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.140463 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.152017 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.151984 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.152142 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.152128 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6vmw\" (UniqueName: \"kubernetes.io/projected/d0ed161a-7533-42b6-94c9-bfaacc784178-kube-api-access-b6vmw\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.167562 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.167538 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:53.200057 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.200022 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v"] Nov 02 20:37:53.296858 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.296821 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a6c564-cef2-4570-b7d9-744b614c39d6-istiod-ca-cert\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.297048 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.296969 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jnmd\" (UniqueName: \"kubernetes.io/projected/04a6c564-cef2-4570-b7d9-744b614c39d6-kube-api-access-7jnmd\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.297048 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.297022 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-envoy\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.297183 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.297123 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-workload-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.297183 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.297157 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-credential-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.297318 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.297304 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-data\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.297374 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.297328 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-workload-certs\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.297446 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.297428 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-podinfo\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.297500 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.297487 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-token\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.398889 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.398804 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-podinfo\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.398889 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.398863 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-token\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.399178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.398919 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a6c564-cef2-4570-b7d9-744b614c39d6-istiod-ca-cert\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.399178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.398966 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jnmd\" (UniqueName: \"kubernetes.io/projected/04a6c564-cef2-4570-b7d9-744b614c39d6-kube-api-access-7jnmd\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.399178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.398999 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-envoy\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.399178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.399020 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-workload-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.399178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.399045 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-credential-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.399178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.399170 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-data\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.399505 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.399200 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-workload-certs\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.399569 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.399547 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-credential-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.399694 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.399640 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-data\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.399694 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.399670 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-workload-certs\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.399920 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.399803 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-workload-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.399920 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.399888 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a6c564-cef2-4570-b7d9-744b614c39d6-istiod-ca-cert\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.401499 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.401481 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-envoy\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.401635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.401620 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-podinfo\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.456980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.456953 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-token\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.457154 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.457136 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jnmd\" (UniqueName: \"kubernetes.io/projected/04a6c564-cef2-4570-b7d9-744b614c39d6-kube-api-access-7jnmd\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.457914 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.457898 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:53.612557 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:37:53.609043 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ed161a_7533_42b6_94c9_bfaacc784178.slice/crio-d1e265de2b87e66f8546de88dcfe47aae40deff4b8c8ab5e721a801b089af41a WatchSource:0}: Error finding container d1e265de2b87e66f8546de88dcfe47aae40deff4b8c8ab5e721a801b089af41a: Status 404 returned error can't find the container with id d1e265de2b87e66f8546de88dcfe47aae40deff4b8c8ab5e721a801b089af41a Nov 02 20:37:53.612557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.611718 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:53.612557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.611813 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:53.612557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.611855 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:53.613455 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.613427 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6"] Nov 02 20:37:53.724717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.724678 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:53.850452 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.850417 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v"] Nov 02 20:37:53.857219 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.856797 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:53.857219 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.856950 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:53.857219 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:53.857061 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:37:54.458305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:54.458258 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" event={"ID":"04a6c564-cef2-4570-b7d9-744b614c39d6","Type":"ContainerStarted","Data":"e6c912fb3f87832a7e43d5d546b1cf99396a4a248d03a316e7dc0503ac8c210d"} Nov 02 20:37:54.458485 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:54.458328 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" event={"ID":"04a6c564-cef2-4570-b7d9-744b614c39d6","Type":"ContainerStarted","Data":"4ae737f4beef8f14288b39c9a45c2feac34c085a8115c477036840a306d196cd"} Nov 02 20:37:54.460324 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:54.460292 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" event={"ID":"d0ed161a-7533-42b6-94c9-bfaacc784178","Type":"ContainerStarted","Data":"68272fa1c56e7e84119fadcb276bd18068ba62d38592cc42ea4fac96a4f0c0d0"} Nov 02 20:37:54.460455 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:54.460345 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" event={"ID":"d0ed161a-7533-42b6-94c9-bfaacc784178","Type":"ContainerStarted","Data":"d1e265de2b87e66f8546de88dcfe47aae40deff4b8c8ab5e721a801b089af41a"} Nov 02 20:37:54.587107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:54.586576 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" podStartSLOduration=1.5865618449999999 podStartE2EDuration="1.586561845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:54.585909218 +0000 UTC m=+1402.875565746" watchObservedRunningTime="2026-11-02 20:37:54.586561845 +0000 UTC m=+1402.876218373" Nov 02 20:37:54.711793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:54.711687 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" podStartSLOduration=2.711672486 podStartE2EDuration="2.711672486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:54.706008597 +0000 UTC m=+1402.995665125" watchObservedRunningTime="2026-11-02 20:37:54.711672486 +0000 UTC m=+1403.001329063" Nov 02 20:37:54.879377 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:54.879345 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28"] Nov 02 20:37:54.885367 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:54.885323 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" Nov 02 20:37:54.930107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:54.922348 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28"] Nov 02 20:37:54.978212 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:54.977686 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:37:54.978212 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:54.977812 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:37:55.079113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.076756 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39d28439-vhd28\" (UID: \"5989a61f-f5ac-4ab2-8888-9f6568b02ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" Nov 02 20:37:55.079113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.076828 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39d28439-vhd28\" (UID: \"5989a61f-f5ac-4ab2-8888-9f6568b02ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" Nov 02 20:37:55.079113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.076992 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx6pc\" (UniqueName: \"kubernetes.io/projected/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-kube-api-access-fx6pc\") pod \"maas-api-revoke-sub-39d28439-vhd28\" (UID: \"5989a61f-f5ac-4ab2-8888-9f6568b02ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" Nov 02 20:37:55.174728 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.170159 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:37:55.174728 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.174310 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 20:37:55.174728 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.174365 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" podUID="d0ed161a-7533-42b6-94c9-bfaacc784178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 20:37:55.178318 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.178288 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39d28439-vhd28\" (UID: \"5989a61f-f5ac-4ab2-8888-9f6568b02ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" Nov 02 20:37:55.178580 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.178560 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx6pc\" (UniqueName: \"kubernetes.io/projected/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-kube-api-access-fx6pc\") pod \"maas-api-revoke-sub-39d28439-vhd28\" (UID: \"5989a61f-f5ac-4ab2-8888-9f6568b02ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" Nov 02 20:37:55.179003 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.178980 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39d28439-vhd28\" (UID: \"5989a61f-f5ac-4ab2-8888-9f6568b02ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" Nov 02 20:37:55.179214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.179155 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39d28439-vhd28\" (UID: \"5989a61f-f5ac-4ab2-8888-9f6568b02ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" Nov 02 20:37:55.181848 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.181825 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39d28439-vhd28\" (UID: \"5989a61f-f5ac-4ab2-8888-9f6568b02ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" Nov 02 20:37:55.273156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.273020 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx6pc\" (UniqueName: \"kubernetes.io/projected/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-kube-api-access-fx6pc\") pod \"maas-api-revoke-sub-39d28439-vhd28\" (UID: \"5989a61f-f5ac-4ab2-8888-9f6568b02ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" Nov 02 20:37:55.462110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.461541 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:37:55.471108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.469224 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 20:37:55.471108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.469295 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" podUID="04a6c564-cef2-4570-b7d9-744b614c39d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 20:37:55.500418 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.500374 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" Nov 02 20:37:55.980922 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:55.980888 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28"] Nov 02 20:37:56.175256 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:56.171413 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 20:37:56.175256 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:56.171477 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" podUID="d0ed161a-7533-42b6-94c9-bfaacc784178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 20:37:56.467217 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:56.467183 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 20:37:56.467521 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:56.467490 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" podUID="04a6c564-cef2-4570-b7d9-744b614c39d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 20:37:56.495871 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:56.495839 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" event={"ID":"5989a61f-f5ac-4ab2-8888-9f6568b02ee9","Type":"ContainerStarted","Data":"5ca7fdce2b98339b668b8e48535d74fcb4a090d18cd87141d8ea9b263a3207ed"} Nov 02 20:37:56.607114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:56.607044 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" podStartSLOduration=2.607029551 podStartE2EDuration="2.607029551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:56.606309232 +0000 UTC m=+1404.895965761" watchObservedRunningTime="2026-11-02 20:37:56.607029551 +0000 UTC m=+1404.896686079" Nov 02 20:37:57.172137 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:57.171434 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 20:37:57.172137 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:57.171496 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" podUID="d0ed161a-7533-42b6-94c9-bfaacc784178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 20:37:57.462431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:57.461668 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 20:37:57.462431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:57.461725 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" podUID="04a6c564-cef2-4570-b7d9-744b614c39d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 20:37:57.506325 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:57.505997 4971 generic.go:356] "Generic (PLEG): container finished" podID="5989a61f-f5ac-4ab2-8888-9f6568b02ee9" containerID="18dcb6c6b33845851164122a39c2e41267c0c23c6df8e4a260ad1fd6bd0e15b5" exitCode=0 Nov 02 20:37:57.506325 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:57.506103 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" event={"ID":"5989a61f-f5ac-4ab2-8888-9f6568b02ee9","Type":"ContainerDied","Data":"18dcb6c6b33845851164122a39c2e41267c0c23c6df8e4a260ad1fd6bd0e15b5"} Nov 02 20:37:57.580279 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:57.580247 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:58.173210 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:58.172618 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 20:37:58.173210 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:58.172678 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" podUID="d0ed161a-7533-42b6-94c9-bfaacc784178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 20:37:58.470167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:58.463525 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 20:37:58.470167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:58.463589 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" podUID="04a6c564-cef2-4570-b7d9-744b614c39d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 20:37:58.909001 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:58.908805 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:37:58.921399 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:58.921366 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" Nov 02 20:37:59.102267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:59.102235 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-maas-api-openshift-ca\") pod \"5989a61f-f5ac-4ab2-8888-9f6568b02ee9\" (UID: \"5989a61f-f5ac-4ab2-8888-9f6568b02ee9\") " Nov 02 20:37:59.102479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:59.102407 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-maas-api-xks-ca\") pod \"5989a61f-f5ac-4ab2-8888-9f6568b02ee9\" (UID: \"5989a61f-f5ac-4ab2-8888-9f6568b02ee9\") " Nov 02 20:37:59.102479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:59.102448 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-kube-api-access-fx6pc\" (UniqueName: \"kubernetes.io/projected/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-kube-api-access-fx6pc\") pod \"5989a61f-f5ac-4ab2-8888-9f6568b02ee9\" (UID: \"5989a61f-f5ac-4ab2-8888-9f6568b02ee9\") " Nov 02 20:37:59.102656 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:59.102626 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-maas-api-openshift-ca" pod "5989a61f-f5ac-4ab2-8888-9f6568b02ee9" (UID: "5989a61f-f5ac-4ab2-8888-9f6568b02ee9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:59.103445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:59.103418 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:59.104880 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:59.104850 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-maas-api-xks-ca" pod "5989a61f-f5ac-4ab2-8888-9f6568b02ee9" (UID: "5989a61f-f5ac-4ab2-8888-9f6568b02ee9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:59.105308 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:59.105262 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-kube-api-access-fx6pc" pod "5989a61f-f5ac-4ab2-8888-9f6568b02ee9" (UID: "5989a61f-f5ac-4ab2-8888-9f6568b02ee9"). InnerVolumeSpecName "kube-api-access-fx6pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:59.179487 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:59.179454 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 20:37:59.180060 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:59.180030 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" podUID="d0ed161a-7533-42b6-94c9-bfaacc784178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 20:37:59.204897 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:59.204867 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:59.204897 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:59.204898 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx6pc\" (UniqueName: \"kubernetes.io/projected/5989a61f-f5ac-4ab2-8888-9f6568b02ee9-kube-api-access-fx6pc\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:59.462168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:59.461423 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 20:37:59.462168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:59.461483 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" podUID="04a6c564-cef2-4570-b7d9-744b614c39d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 20:37:59.572131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:59.554654 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" event={"ID":"5989a61f-f5ac-4ab2-8888-9f6568b02ee9","Type":"ContainerDied","Data":"5ca7fdce2b98339b668b8e48535d74fcb4a090d18cd87141d8ea9b263a3207ed"} Nov 02 20:37:59.572131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:59.554716 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ca7fdce2b98339b668b8e48535d74fcb4a090d18cd87141d8ea9b263a3207ed" Nov 02 20:37:59.572131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:37:59.554786 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28" Nov 02 20:37:59.601902 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:37:59.601869 4971 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-conmon-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5989a61f_f5ac_4ab2_8888_9f6568b02ee9.slice/crio-5ca7fdce2b98339b668b8e48535d74fcb4a090d18cd87141d8ea9b263a3207ed\": RecentStats: unable to find data in memory cache]" Nov 02 20:38:00.168170 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:00.168130 4971 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 20:38:00.168353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:00.168202 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" podUID="d0ed161a-7533-42b6-94c9-bfaacc784178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 20:38:00.459224 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:00.459128 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 20:38:00.459224 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:00.459193 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" podUID="04a6c564-cef2-4570-b7d9-744b614c39d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 20:38:01.173133 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.173066 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:38:01.173558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.173540 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:38:01.174282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.174259 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:38:01.465653 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.464489 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:38:01.466135 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.465917 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:38:01.469296 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.469271 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:38:01.708713 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.708679 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q"] Nov 02 20:38:01.709510 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.709486 4971 cpu_manager.go:414] "containerMap: removing container" podUID="5989a61f-f5ac-4ab2-8888-9f6568b02ee9" containerName="revoke-keys" Nov 02 20:38:01.709510 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.709506 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5989a61f-f5ac-4ab2-8888-9f6568b02ee9" containerName="revoke-keys" Nov 02 20:38:01.710297 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.709615 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="5989a61f-f5ac-4ab2-8888-9f6568b02ee9" containerName="revoke-keys" Nov 02 20:38:01.714196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.714164 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.750446 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.750417 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5"] Nov 02 20:38:01.759420 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.759394 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.772931 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.772903 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q"] Nov 02 20:38:01.810042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.810015 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5"] Nov 02 20:38:01.844668 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.844635 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.845201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.844694 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.845201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.844723 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.845201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.844757 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.845201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.844795 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.845201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.844820 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.845201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.844955 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.845201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.845112 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.845201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.845191 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxkdv\" (UniqueName: \"kubernetes.io/projected/75f18eb1-5617-4f8a-b812-b0b9abe2078e-kube-api-access-cxkdv\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.845680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.845236 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.845680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.845306 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.845680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.845338 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.845680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.845413 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.845680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.845458 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.845680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.845527 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.845680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.845637 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.846053 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.845729 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gspl\" (UniqueName: \"kubernetes.io/projected/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-kube-api-access-2gspl\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.846053 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.845756 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.946925 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.946883 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.947160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947010 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.947160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947121 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gspl\" (UniqueName: \"kubernetes.io/projected/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-kube-api-access-2gspl\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.947160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947154 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.947369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947211 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.947369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947249 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.947369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947290 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.947369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947323 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.947369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947359 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.947632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947383 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.947632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947433 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.947632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947505 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.947632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947553 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxkdv\" (UniqueName: \"kubernetes.io/projected/75f18eb1-5617-4f8a-b812-b0b9abe2078e-kube-api-access-cxkdv\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.947632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947258 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.947632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947578 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.947632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947583 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.947632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947599 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.948235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947659 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.948235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947690 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.948235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947741 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.948235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947763 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.948235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947790 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.948235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.947870 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.948235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.948196 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.948235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.948224 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.948663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.948580 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.948663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.948613 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.948881 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.948855 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.949909 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.949883 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.950646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.950626 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:01.950930 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.950907 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:01.951037 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:01.950967 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:02.007664 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.007590 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:02.009407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.009381 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxkdv\" (UniqueName: \"kubernetes.io/projected/75f18eb1-5617-4f8a-b812-b0b9abe2078e-kube-api-access-cxkdv\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:02.010899 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.010875 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gspl\" (UniqueName: \"kubernetes.io/projected/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-kube-api-access-2gspl\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:02.014452 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.014431 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:02.039434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.039407 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:02.085588 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.085559 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:02.496923 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:38:02.496893 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda416fb5c_b8c6_4159_9cc3_d2f6ea601b50.slice/crio-4261490399a8a6f854146b881dc4ac7f177290e20bb2d8ba7afd74efdb422280 WatchSource:0}: Error finding container 4261490399a8a6f854146b881dc4ac7f177290e20bb2d8ba7afd74efdb422280: Status 404 returned error can't find the container with id 4261490399a8a6f854146b881dc4ac7f177290e20bb2d8ba7afd74efdb422280 Nov 02 20:38:02.498911 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.498865 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q"] Nov 02 20:38:02.499847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.499809 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:02.499945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.499877 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:02.499945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.499911 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:02.546118 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:38:02.546050 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f18eb1_5617_4f8a_b812_b0b9abe2078e.slice/crio-04df6782884d84390ecff7f2b76d7f543f68a9ca823068202d198b2f859f18d8 WatchSource:0}: Error finding container 04df6782884d84390ecff7f2b76d7f543f68a9ca823068202d198b2f859f18d8: Status 404 returned error can't find the container with id 04df6782884d84390ecff7f2b76d7f543f68a9ca823068202d198b2f859f18d8 Nov 02 20:38:02.549187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.549148 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:02.549307 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.549237 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:02.549307 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.549281 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:02.550226 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.550203 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5"] Nov 02 20:38:02.595681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.595654 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" event={"ID":"75f18eb1-5617-4f8a-b812-b0b9abe2078e","Type":"ContainerStarted","Data":"04df6782884d84390ecff7f2b76d7f543f68a9ca823068202d198b2f859f18d8"} Nov 02 20:38:02.597608 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.597578 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" event={"ID":"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50","Type":"ContainerStarted","Data":"9dfcdf49d8bd7a38bf26c5ebb23536c4f65b58c59775bde471bdc699f723866e"} Nov 02 20:38:02.597778 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.597631 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" event={"ID":"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50","Type":"ContainerStarted","Data":"4261490399a8a6f854146b881dc4ac7f177290e20bb2d8ba7afd74efdb422280"} Nov 02 20:38:02.720249 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:02.720191 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" podStartSLOduration=1.72017519 podStartE2EDuration="1.72017519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:02.71220613 +0000 UTC m=+1411.001862656" watchObservedRunningTime="2026-11-02 20:38:02.72017519 +0000 UTC m=+1411.009831717" Nov 02 20:38:03.039922 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:03.039888 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:03.041163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:03.041137 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:38:03.041273 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:03.041196 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:38:03.348862 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:38:03.348818 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:38:03.608483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:03.608391 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" event={"ID":"75f18eb1-5617-4f8a-b812-b0b9abe2078e","Type":"ContainerStarted","Data":"6b15df5f0d893ce8967faef193df7e723cd273fd027ffc6d97e0e70d3ea7aa6f"} Nov 02 20:38:03.700356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:03.700324 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:03.814950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:03.814887 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" podStartSLOduration=2.814869089 podStartE2EDuration="2.814869089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:03.808445197 +0000 UTC m=+1412.098101748" watchObservedRunningTime="2026-11-02 20:38:03.814869089 +0000 UTC m=+1412.104525618" Nov 02 20:38:03.970602 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:03.970509 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:04.041873 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:04.041837 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:38:04.042202 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:04.042160 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:38:04.086256 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:04.086227 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:04.088281 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:04.088262 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 20:38:04.088436 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:04.088416 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 20:38:05.057111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:05.056369 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:38:05.057111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:05.056427 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:38:05.086723 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:05.086690 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 20:38:05.086983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:05.086757 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 20:38:05.508110 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:38:05.503504 4971 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-conmon-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:38:06.092119 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:06.084804 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:38:06.092119 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:06.084862 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:38:06.092119 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:06.086960 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 20:38:06.092119 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:06.087004 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 20:38:06.447108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:06.443183 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:06.469109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:06.466176 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:06.723803 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:06.723703 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:06.725153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:06.724030 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:06.790467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:06.744021 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:06.791142 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:06.765615 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:06.791409 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:06.784209 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:07.003162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:06.998108 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:07.098121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:07.096222 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:38:07.098121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:07.096305 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:38:07.098121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:07.097022 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 20:38:07.098121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:07.097060 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 20:38:07.180353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:07.177015 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:07.197309 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:07.197260 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:07.450846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:07.448269 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:07.450846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:07.449055 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:38:07.467720 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:07.467679 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:07.468205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:07.468179 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:38:07.523195 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:07.523155 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:08.059439 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.059389 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 02 20:38:08.060558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.060516 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:08.141784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.111580 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 02 20:38:08.141784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.111827 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 02 20:38:08.141784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.111996 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:08.141784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.120959 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 02 20:38:08.141784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.123262 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:38:08.141784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.123308 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:38:08.141784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.123356 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": context deadline exceeded" Nov 02 20:38:08.141784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.123551 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 20:38:08.141784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.123578 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 20:38:08.226984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.199723 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:08.226984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.199991 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 20:38:08.353477 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.353444 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:38:08.475960 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.475931 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:38:08.476261 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.476244 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:38:08.476432 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.476419 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:38:08.476581 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.476567 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:38:08.476716 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.476705 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:38:08.476850 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.476838 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:38:08.476983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.476972 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:38:08.492110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.354597 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:08.495303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.469655 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded" Nov 02 20:38:08.495502 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.448714 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:08.495645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.469620 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:08.549152 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.523777 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 02 20:38:08.549707 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.549663 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:38:08.594418 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.594385 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8"] Nov 02 20:38:08.739165 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.737347 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:08.898726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.857257 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-workload-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:08.898726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.857304 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-token\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:08.898726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.857389 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-workload-certs\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:08.898726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.857560 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g42d8\" (UniqueName: \"kubernetes.io/projected/cb7f2447-9d5a-4901-865e-9f61bff9b740-kube-api-access-g42d8\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:08.898726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.857594 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb7f2447-9d5a-4901-865e-9f61bff9b740-istiod-ca-cert\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:08.898726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.857703 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-credential-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:08.898726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.857778 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-podinfo\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:08.898726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.857838 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-envoy\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:08.898726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:08.857860 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-data\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.060162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.039942 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:38:09.089185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.089143 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:38:09.089423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.088175 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:09.089910 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.055393 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-credential-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.194115 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.190288 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-podinfo\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.227793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.227761 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-envoy\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.241262 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.241227 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-data\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.241506 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.134730 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:09.247302 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.134773 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.167183 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.088613 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.270513 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.202997 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.202957 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.203664 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-credential-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.134634 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.203717 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/health\": context deadline exceeded" Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.240736 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-data\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.271609 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-workload-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.271648 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-token\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.271747 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-workload-certs\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.271871 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g42d8\" (UniqueName: \"kubernetes.io/projected/cb7f2447-9d5a-4901-865e-9f61bff9b740-kube-api-access-g42d8\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.271905 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb7f2447-9d5a-4901-865e-9f61bff9b740-istiod-ca-cert\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.272939 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb7f2447-9d5a-4901-865e-9f61bff9b740-istiod-ca-cert\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.273229 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-workload-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.283132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.273782 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-workload-certs\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.319468 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.134705 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:09.366108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.357610 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8"] Nov 02 20:38:09.427134 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.423473 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-envoy\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.442124 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.436158 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-token\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.564110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.528726 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 20:38:09.564110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.529156 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 02 20:38:09.603289 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.540313 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g42d8\" (UniqueName: \"kubernetes.io/projected/cb7f2447-9d5a-4901-865e-9f61bff9b740-kube-api-access-g42d8\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.603587 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.603227 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 02 20:38:09.680124 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.678539 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-podinfo\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:09.883158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:09.871363 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:10.014972 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.013693 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:10.105681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.105638 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 02 20:38:10.272636 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.272530 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:10.274471 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.274434 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:10.276214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.276184 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:10.277456 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.276173 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:10.289061 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.276219 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 20:38:10.336108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.276249 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:10.336108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.292151 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 20:38:10.336108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.324369 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 20:38:10.336108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.322510 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:10.336108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.301435 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:10.374252 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.374219 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:38:10.374515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.374485 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:38:10.551879 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.551253 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 20:38:10.588145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.575756 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:10.633934 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:10.632577 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 02 20:38:11.096119 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:11.090055 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:38:11.096119 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:11.090145 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:38:11.096119 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:11.090274 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 20:38:11.096119 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:11.090303 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 20:38:11.136123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:11.134545 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 02 20:38:11.149688 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:11.149014 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:11.276481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:11.276430 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 02 20:38:11.279805 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:11.279766 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:11.298273 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:11.284255 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:11.298273 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:11.285012 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 02 20:38:11.335599 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:11.335553 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:11.359338 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:11.336168 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 02 20:38:11.359967 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:11.324732 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 20:38:11.601342 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:11.599178 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 20:38:11.601342 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:11.600871 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:11.640845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:11.633355 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:12.167034 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.163414 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:12.187055 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.186935 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:38:12.187055 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.186995 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:38:12.319044 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.312623 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 20:38:12.345168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.320544 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 02 20:38:12.345168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.324050 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 20:38:12.345168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.325010 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 02 20:38:12.386107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.345596 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:12.386107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.360339 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 20:38:12.386107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.361511 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:12.386107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.378946 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:38:12.386107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.379000 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" podUID="fa9309a3-38ea-4870-91dc-2bbab6b13f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:12.471309 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.471252 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/health\": context deadline exceeded" Nov 02 20:38:12.471773 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.471735 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 20:38:12.635126 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.628055 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 20:38:12.658309 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.658252 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 20:38:12.696298 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.663325 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 02 20:38:12.696298 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.677813 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:12.729576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.729545 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8"] Nov 02 20:38:12.935809 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:12.935778 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:38:13.258533 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.176008 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 02 20:38:13.258533 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.213339 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:13.258533 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.213431 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:13.258533 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.213471 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:13.258533 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:38:13.215155 4971 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-conmon-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:38:13.290015 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.284165 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Nov 02 20:38:13.404260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.404215 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:13.405552 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.405521 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 20:38:13.406189 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.406157 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 02 20:38:13.406912 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.406844 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:13.407563 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.407542 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 02 20:38:13.423097 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.411929 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:38:13.443517 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.430246 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:38:13.449189 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.449165 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:38:13.503981 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.503953 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:38:13.510720 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.510652 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:38:13.560737 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.560697 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 20:38:13.561029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.560996 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 20:38:13.616641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.616617 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:38:13.617605 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.617585 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:38:13.619268 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.619248 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:38:13.620607 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.620585 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:38:13.620764 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.620744 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:38:13.621237 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.621216 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:38:13.684331 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:13.680217 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:38:14.051194 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:14.045033 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:38:14.052228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:14.052188 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:38:14.086294 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:14.086261 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 20:38:14.086589 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:14.086559 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 20:38:14.180240 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:14.180173 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:14.334172 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:14.331970 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" event={"ID":"cb7f2447-9d5a-4901-865e-9f61bff9b740","Type":"ContainerStarted","Data":"33245f28b7fb1691ee59e2b16fbe23f2bee5289eb6413c5ef83e7cea410f0936"} Nov 02 20:38:14.334172 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:14.332042 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" event={"ID":"cb7f2447-9d5a-4901-865e-9f61bff9b740","Type":"ContainerStarted","Data":"beb1cb40ed1d9373832aa0304235f790c87c7c8720db3a6be5947d6db53b7993"} Nov 02 20:38:14.884604 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:14.884568 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:14.885196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:14.884712 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 20:38:14.885196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:14.884758 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" podUID="cb7f2447-9d5a-4901-865e-9f61bff9b740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 20:38:15.008574 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:15.008511 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" podStartSLOduration=7.008489058 podStartE2EDuration="7.008489058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:14.961491492 +0000 UTC m=+1423.251148055" watchObservedRunningTime="2026-11-02 20:38:15.008489058 +0000 UTC m=+1423.298145586" Nov 02 20:38:15.086760 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:15.086714 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 02 20:38:15.086970 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:15.086951 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 02 20:38:15.765353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:15.765319 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:15.766862 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:15.766817 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:15.767730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:15.767714 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:38:15.890400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:15.890368 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 20:38:15.890997 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:15.890954 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" podUID="cb7f2447-9d5a-4901-865e-9f61bff9b740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 20:38:16.903233 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:16.879001 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:16.904641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:16.904194 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 20:38:16.904641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:16.904256 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" podUID="cb7f2447-9d5a-4901-865e-9f61bff9b740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 20:38:17.093561 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:17.093529 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:17.135208 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:17.135173 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:38:17.135467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:17.135438 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:17.154567 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:17.154487 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:17.406900 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:17.406802 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:17.881174 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:17.880889 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 20:38:17.881174 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:17.880963 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" podUID="cb7f2447-9d5a-4901-865e-9f61bff9b740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 20:38:17.904804 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:17.904772 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:17.959484 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:17.959453 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:38:18.251415 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.251325 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk"] Nov 02 20:38:18.264372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.264343 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk" Nov 02 20:38:18.295133 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.280018 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk"] Nov 02 20:38:18.314670 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.314635 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:38:18.314870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.314739 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:38:18.321359 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.321320 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6l7q\" (UniqueName: \"kubernetes.io/projected/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-kube-api-access-z6l7q\") pod \"maas-api-revoke-sub-cf374d6f-mfcxk\" (UID: \"d647dca8-f937-4a30-a7c6-c7039ffbf9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk" Nov 02 20:38:18.321524 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.321405 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf374d6f-mfcxk\" (UID: \"d647dca8-f937-4a30-a7c6-c7039ffbf9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk" Nov 02 20:38:18.321918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.321529 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf374d6f-mfcxk\" (UID: \"d647dca8-f937-4a30-a7c6-c7039ffbf9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk" Nov 02 20:38:18.451117 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.449430 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf374d6f-mfcxk\" (UID: \"d647dca8-f937-4a30-a7c6-c7039ffbf9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk" Nov 02 20:38:18.451117 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.449578 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf374d6f-mfcxk\" (UID: \"d647dca8-f937-4a30-a7c6-c7039ffbf9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk" Nov 02 20:38:18.451117 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.449858 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6l7q\" (UniqueName: \"kubernetes.io/projected/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-kube-api-access-z6l7q\") pod \"maas-api-revoke-sub-cf374d6f-mfcxk\" (UID: \"d647dca8-f937-4a30-a7c6-c7039ffbf9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk" Nov 02 20:38:18.451117 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.450968 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf374d6f-mfcxk\" (UID: \"d647dca8-f937-4a30-a7c6-c7039ffbf9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk" Nov 02 20:38:18.452959 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.452929 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf374d6f-mfcxk\" (UID: \"d647dca8-f937-4a30-a7c6-c7039ffbf9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk" Nov 02 20:38:18.508072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.508006 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6l7q\" (UniqueName: \"kubernetes.io/projected/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-kube-api-access-z6l7q\") pod \"maas-api-revoke-sub-cf374d6f-mfcxk\" (UID: \"d647dca8-f937-4a30-a7c6-c7039ffbf9b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk" Nov 02 20:38:18.596051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.596018 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk" Nov 02 20:38:18.880740 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.880706 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 20:38:18.880917 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:18.880774 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" podUID="cb7f2447-9d5a-4901-865e-9f61bff9b740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 20:38:19.053719 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:38:19.053668 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd647dca8_f937_4a30_a7c6_c7039ffbf9b4.slice/crio-46fb1f84df3105cd68824291bf069d3b21685cd1824d7e8438c5336d7dc7f8f0 WatchSource:0}: Error finding container 46fb1f84df3105cd68824291bf069d3b21685cd1824d7e8438c5336d7dc7f8f0: Status 404 returned error can't find the container with id 46fb1f84df3105cd68824291bf069d3b21685cd1824d7e8438c5336d7dc7f8f0 Nov 02 20:38:19.058270 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:19.058240 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk"] Nov 02 20:38:19.437108 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:38:19.435772 4971 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-conmon-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:38:19.439124 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:38:19.438294 4971 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-conmon-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:38:19.478174 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:19.478140 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk" event={"ID":"d647dca8-f937-4a30-a7c6-c7039ffbf9b4","Type":"ContainerStarted","Data":"46fb1f84df3105cd68824291bf069d3b21685cd1824d7e8438c5336d7dc7f8f0"} Nov 02 20:38:19.880529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:19.880484 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 20:38:19.880739 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:19.880543 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" podUID="cb7f2447-9d5a-4901-865e-9f61bff9b740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 20:38:20.459136 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:38:20.459015 4971 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-conmon-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:38:20.493053 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:20.493015 4971 generic.go:356] "Generic (PLEG): container finished" podID="d647dca8-f937-4a30-a7c6-c7039ffbf9b4" containerID="4bbce4920d6104364166262997e1810dd6ebbd21f65cd07e0025560a805dacfb" exitCode=0 Nov 02 20:38:20.493245 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:20.493139 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk" event={"ID":"d647dca8-f937-4a30-a7c6-c7039ffbf9b4","Type":"ContainerDied","Data":"4bbce4920d6104364166262997e1810dd6ebbd21f65cd07e0025560a805dacfb"} Nov 02 20:38:20.881565 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:20.881528 4971 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 20:38:20.881775 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:20.881609 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" podUID="cb7f2447-9d5a-4901-865e-9f61bff9b740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 20:38:21.746855 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:21.746829 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk" Nov 02 20:38:21.825947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:21.825192 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-maas-api-xks-ca\") pod \"d647dca8-f937-4a30-a7c6-c7039ffbf9b4\" (UID: \"d647dca8-f937-4a30-a7c6-c7039ffbf9b4\") " Nov 02 20:38:21.825947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:21.825265 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-maas-api-openshift-ca\") pod \"d647dca8-f937-4a30-a7c6-c7039ffbf9b4\" (UID: \"d647dca8-f937-4a30-a7c6-c7039ffbf9b4\") " Nov 02 20:38:21.825947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:21.825378 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-kube-api-access-z6l7q\" (UniqueName: \"kubernetes.io/projected/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-kube-api-access-z6l7q\") pod \"d647dca8-f937-4a30-a7c6-c7039ffbf9b4\" (UID: \"d647dca8-f937-4a30-a7c6-c7039ffbf9b4\") " Nov 02 20:38:21.832500 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:21.832460 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-kube-api-access-z6l7q" pod "d647dca8-f937-4a30-a7c6-c7039ffbf9b4" (UID: "d647dca8-f937-4a30-a7c6-c7039ffbf9b4"). InnerVolumeSpecName "kube-api-access-z6l7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:21.833072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:21.832867 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-maas-api-openshift-ca" pod "d647dca8-f937-4a30-a7c6-c7039ffbf9b4" (UID: "d647dca8-f937-4a30-a7c6-c7039ffbf9b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:38:21.836517 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:21.834887 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-maas-api-xks-ca" pod "d647dca8-f937-4a30-a7c6-c7039ffbf9b4" (UID: "d647dca8-f937-4a30-a7c6-c7039ffbf9b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:38:21.845303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:21.845261 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:21.903794 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:21.903761 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:21.905100 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:21.905041 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:21.905846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:21.905825 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:38:21.931697 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:21.931672 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:21.931821 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:21.931699 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:21.931821 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:21.931712 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6l7q\" (UniqueName: \"kubernetes.io/projected/d647dca8-f937-4a30-a7c6-c7039ffbf9b4-kube-api-access-z6l7q\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:22.528300 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:22.528261 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk" Nov 02 20:38:22.528300 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:22.528288 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk" event={"ID":"d647dca8-f937-4a30-a7c6-c7039ffbf9b4","Type":"ContainerDied","Data":"46fb1f84df3105cd68824291bf069d3b21685cd1824d7e8438c5336d7dc7f8f0"} Nov 02 20:38:22.528576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:22.528343 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46fb1f84df3105cd68824291bf069d3b21685cd1824d7e8438c5336d7dc7f8f0" Nov 02 20:38:23.239317 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:38:23.239279 4971 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4d09ac_83dd_4b68_8784_6b5b959498a2.slice/crio-conmon-e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:38:23.412105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.411436 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc"] Nov 02 20:38:23.416101 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.412594 4971 cpu_manager.go:414] "containerMap: removing container" podUID="d647dca8-f937-4a30-a7c6-c7039ffbf9b4" containerName="revoke-keys" Nov 02 20:38:23.416101 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.412622 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d647dca8-f937-4a30-a7c6-c7039ffbf9b4" containerName="revoke-keys" Nov 02 20:38:23.416101 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.412867 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="d647dca8-f937-4a30-a7c6-c7039ffbf9b4" containerName="revoke-keys" Nov 02 20:38:23.421251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.421226 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.447891 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.447421 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m"] Nov 02 20:38:23.458306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.458283 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.476578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.476459 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dg9qz\"" Nov 02 20:38:23.552186 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.551170 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc"] Nov 02 20:38:23.583208 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.583163 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.583413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.583257 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.583413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.583342 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.583413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.583374 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27615c4a-c472-4862-b72d-95dff20b301f-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.583413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.583410 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hjb5\" (UniqueName: \"kubernetes.io/projected/35418b46-29f4-48ba-a0ac-8ca3243c4152-kube-api-access-4hjb5\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.583631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.583449 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.583631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.583521 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.583631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.583555 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27615c4a-c472-4862-b72d-95dff20b301f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.583783 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.583651 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35418b46-29f4-48ba-a0ac-8ca3243c4152-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.583783 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.583682 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.583783 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.583711 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.583783 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.583747 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.583995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.583782 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.583995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.583829 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh4bs\" (UniqueName: \"kubernetes.io/projected/27615c4a-c472-4862-b72d-95dff20b301f-kube-api-access-rh4bs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.583995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.583875 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.583995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.583972 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.584243 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.584141 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27615c4a-c472-4862-b72d-95dff20b301f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.584243 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.584185 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.623153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.621459 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m"] Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.742779 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27615c4a-c472-4862-b72d-95dff20b301f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.742841 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.742879 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.742929 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.743015 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.743046 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27615c4a-c472-4862-b72d-95dff20b301f-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.743073 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hjb5\" (UniqueName: \"kubernetes.io/projected/35418b46-29f4-48ba-a0ac-8ca3243c4152-kube-api-access-4hjb5\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.743124 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.743189 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.743218 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27615c4a-c472-4862-b72d-95dff20b301f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.743311 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35418b46-29f4-48ba-a0ac-8ca3243c4152-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.743335 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.743361 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.743407 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.743443 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.743491 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh4bs\" (UniqueName: \"kubernetes.io/projected/27615c4a-c472-4862-b72d-95dff20b301f-kube-api-access-rh4bs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.743528 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.743619 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.744728 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27615c4a-c472-4862-b72d-95dff20b301f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.745051 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.752941 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35418b46-29f4-48ba-a0ac-8ca3243c4152-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.769160 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.783103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.770138 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.830509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.801546 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.830509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.801780 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.830509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.801981 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.830509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.802434 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.830509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.802960 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.830509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.804186 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.830509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.804464 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.830509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.815634 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27615c4a-c472-4862-b72d-95dff20b301f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.885513 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.885471 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hjb5\" (UniqueName: \"kubernetes.io/projected/35418b46-29f4-48ba-a0ac-8ca3243c4152-kube-api-access-4hjb5\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.885670 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.885521 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27615c4a-c472-4862-b72d-95dff20b301f-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.888109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.888063 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:23.888281 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.888258 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh4bs\" (UniqueName: \"kubernetes.io/projected/27615c4a-c472-4862-b72d-95dff20b301f-kube-api-access-rh4bs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-z6cnc\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:23.889499 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:23.889473 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vxc8m\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:24.078496 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.078459 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:24.110731 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.110698 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:24.398329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.398293 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:24.579301 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:38:24.579268 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27615c4a_c472_4862_b72d_95dff20b301f.slice/crio-91dc11f0f74ce805ab9417b29952901a1b92d30002e23fdac771074dc30bacd3 WatchSource:0}: Error finding container 91dc11f0f74ce805ab9417b29952901a1b92d30002e23fdac771074dc30bacd3: Status 404 returned error can't find the container with id 91dc11f0f74ce805ab9417b29952901a1b92d30002e23fdac771074dc30bacd3 Nov 02 20:38:24.579904 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:38:24.579881 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35418b46_29f4_48ba_a0ac_8ca3243c4152.slice/crio-df24dc8952f16bf6c9458c85f676be2e367e62ee69e94b410722950266e9cec3 WatchSource:0}: Error finding container df24dc8952f16bf6c9458c85f676be2e367e62ee69e94b410722950266e9cec3: Status 404 returned error can't find the container with id df24dc8952f16bf6c9458c85f676be2e367e62ee69e94b410722950266e9cec3 Nov 02 20:38:24.582396 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.582366 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc"] Nov 02 20:38:24.582596 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.582555 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:24.582648 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.582603 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:24.582648 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.582635 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:24.582716 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.582680 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:24.582716 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.582684 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:24.582786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.582732 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:24.591150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.591131 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m"] Nov 02 20:38:24.647968 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.647944 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:24.680309 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.680277 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" event={"ID":"27615c4a-c472-4862-b72d-95dff20b301f","Type":"ContainerStarted","Data":"1a4dec54642036f975580a53caff9a2d729d01cd3477f96e9890bfb21830ea3f"} Nov 02 20:38:24.680465 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.680329 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" event={"ID":"27615c4a-c472-4862-b72d-95dff20b301f","Type":"ContainerStarted","Data":"91dc11f0f74ce805ab9417b29952901a1b92d30002e23fdac771074dc30bacd3"} Nov 02 20:38:24.690563 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.689915 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" event={"ID":"35418b46-29f4-48ba-a0ac-8ca3243c4152","Type":"ContainerStarted","Data":"2247131d43df52af6933e053c9c7e134126fa355d931794b1acca7310c875591"} Nov 02 20:38:24.690563 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.689975 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" event={"ID":"35418b46-29f4-48ba-a0ac-8ca3243c4152","Type":"ContainerStarted","Data":"df24dc8952f16bf6c9458c85f676be2e367e62ee69e94b410722950266e9cec3"} Nov 02 20:38:24.801198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.801146 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" podStartSLOduration=1.801132726 podStartE2EDuration="1.801132726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:24.794807365 +0000 UTC m=+1433.084463894" watchObservedRunningTime="2026-11-02 20:38:24.801132726 +0000 UTC m=+1433.090789254" Nov 02 20:38:24.913226 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:24.913109 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" podStartSLOduration=1.913075329 podStartE2EDuration="1.913075329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:24.907745112 +0000 UTC m=+1433.197401679" watchObservedRunningTime="2026-11-02 20:38:24.913075329 +0000 UTC m=+1433.202731926" Nov 02 20:38:25.078907 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:25.078877 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:25.080727 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:25.080690 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-z6cnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 20:38:25.080881 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:25.080750 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" podUID="27615c4a-c472-4862-b72d-95dff20b301f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 20:38:25.111209 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:25.111169 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:25.112734 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:25.112708 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vxc8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:38:25.112972 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:25.112775 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" podUID="35418b46-29f4-48ba-a0ac-8ca3243c4152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 20:38:26.081224 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:26.080652 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-z6cnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 20:38:26.081224 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:26.080712 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" podUID="27615c4a-c472-4862-b72d-95dff20b301f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 20:38:26.111531 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:26.111500 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vxc8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:38:26.111671 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:26.111543 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" podUID="35418b46-29f4-48ba-a0ac-8ca3243c4152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 20:38:26.744113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:26.741642 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:27.080159 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:27.079573 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-z6cnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 20:38:27.080159 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:27.080046 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" podUID="27615c4a-c472-4862-b72d-95dff20b301f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 20:38:27.111284 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:27.111260 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vxc8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:38:27.111647 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:27.111300 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" podUID="35418b46-29f4-48ba-a0ac-8ca3243c4152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 20:38:28.080345 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:28.079681 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-z6cnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 20:38:28.080345 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:28.079757 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" podUID="27615c4a-c472-4862-b72d-95dff20b301f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 20:38:28.111706 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:28.111676 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vxc8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:38:28.112067 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:28.111728 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" podUID="35418b46-29f4-48ba-a0ac-8ca3243c4152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 20:38:29.045106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.044783 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q"] Nov 02 20:38:29.077476 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.077383 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" Nov 02 20:38:29.079830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.079802 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-z6cnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 20:38:29.080058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.080034 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" podUID="27615c4a-c472-4862-b72d-95dff20b301f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 20:38:29.087604 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.087575 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q"] Nov 02 20:38:29.111648 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.111624 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vxc8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:38:29.111800 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.111664 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" podUID="35418b46-29f4-48ba-a0ac-8ca3243c4152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 20:38:29.193329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.193296 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:38:29.193502 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.193460 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:38:29.210047 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.210011 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7vwl\" (UniqueName: \"kubernetes.io/projected/508e995b-a379-439b-a5a1-92e2d46214f8-kube-api-access-j7vwl\") pod \"maas-api-revoke-sub-a75f6cce-v626q\" (UID: \"508e995b-a379-439b-a5a1-92e2d46214f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" Nov 02 20:38:29.210305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.210284 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/508e995b-a379-439b-a5a1-92e2d46214f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a75f6cce-v626q\" (UID: \"508e995b-a379-439b-a5a1-92e2d46214f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" Nov 02 20:38:29.210385 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.210350 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/508e995b-a379-439b-a5a1-92e2d46214f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a75f6cce-v626q\" (UID: \"508e995b-a379-439b-a5a1-92e2d46214f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" Nov 02 20:38:29.311534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.311449 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/508e995b-a379-439b-a5a1-92e2d46214f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a75f6cce-v626q\" (UID: \"508e995b-a379-439b-a5a1-92e2d46214f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" Nov 02 20:38:29.311534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.311501 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/508e995b-a379-439b-a5a1-92e2d46214f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a75f6cce-v626q\" (UID: \"508e995b-a379-439b-a5a1-92e2d46214f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" Nov 02 20:38:29.311757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.311635 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7vwl\" (UniqueName: \"kubernetes.io/projected/508e995b-a379-439b-a5a1-92e2d46214f8-kube-api-access-j7vwl\") pod \"maas-api-revoke-sub-a75f6cce-v626q\" (UID: \"508e995b-a379-439b-a5a1-92e2d46214f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" Nov 02 20:38:29.312455 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.312430 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/508e995b-a379-439b-a5a1-92e2d46214f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a75f6cce-v626q\" (UID: \"508e995b-a379-439b-a5a1-92e2d46214f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" Nov 02 20:38:29.315616 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.315593 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/508e995b-a379-439b-a5a1-92e2d46214f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a75f6cce-v626q\" (UID: \"508e995b-a379-439b-a5a1-92e2d46214f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" Nov 02 20:38:29.379180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.375780 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7vwl\" (UniqueName: \"kubernetes.io/projected/508e995b-a379-439b-a5a1-92e2d46214f8-kube-api-access-j7vwl\") pod \"maas-api-revoke-sub-a75f6cce-v626q\" (UID: \"508e995b-a379-439b-a5a1-92e2d46214f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" Nov 02 20:38:29.398544 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.398514 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" Nov 02 20:38:29.895058 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:38:29.895029 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508e995b_a379_439b_a5a1_92e2d46214f8.slice/crio-e486b741a4de83b8f04fc0f2639176eda3d533f2d14bf37392a784dba406e947 WatchSource:0}: Error finding container e486b741a4de83b8f04fc0f2639176eda3d533f2d14bf37392a784dba406e947: Status 404 returned error can't find the container with id e486b741a4de83b8f04fc0f2639176eda3d533f2d14bf37392a784dba406e947 Nov 02 20:38:29.898801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.898763 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q"] Nov 02 20:38:29.959106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:29.950957 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:30.080146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:30.079617 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-z6cnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 20:38:30.080146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:30.079677 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" podUID="27615c4a-c472-4862-b72d-95dff20b301f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 20:38:30.111356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:30.111247 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vxc8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:38:30.111356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:30.111295 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" podUID="35418b46-29f4-48ba-a0ac-8ca3243c4152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 20:38:30.812220 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:30.812172 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" event={"ID":"508e995b-a379-439b-a5a1-92e2d46214f8","Type":"ContainerStarted","Data":"9e6ceb07170bbea48d266658a2d690c5ce86097fcfd51c3d04fdcb0adb1c3ca9"} Nov 02 20:38:30.812902 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:30.812267 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" event={"ID":"508e995b-a379-439b-a5a1-92e2d46214f8","Type":"ContainerStarted","Data":"e486b741a4de83b8f04fc0f2639176eda3d533f2d14bf37392a784dba406e947"} Nov 02 20:38:30.924300 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:30.924247 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" podStartSLOduration=2.924231587 podStartE2EDuration="2.924231587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:30.921804112 +0000 UTC m=+1439.211460644" watchObservedRunningTime="2026-11-02 20:38:30.924231587 +0000 UTC m=+1439.213888114" Nov 02 20:38:31.080052 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:31.079957 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-z6cnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 20:38:31.080052 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:31.080032 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" podUID="27615c4a-c472-4862-b72d-95dff20b301f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 20:38:31.111307 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:31.111278 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vxc8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:38:31.111408 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:31.111339 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" podUID="35418b46-29f4-48ba-a0ac-8ca3243c4152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 20:38:31.546946 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:31.546275 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:31.804786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:31.804680 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:31.842055 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:31.842014 4971 generic.go:356] "Generic (PLEG): container finished" podID="508e995b-a379-439b-a5a1-92e2d46214f8" containerID="9e6ceb07170bbea48d266658a2d690c5ce86097fcfd51c3d04fdcb0adb1c3ca9" exitCode=0 Nov 02 20:38:31.842523 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:31.842162 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" event={"ID":"508e995b-a379-439b-a5a1-92e2d46214f8","Type":"ContainerDied","Data":"9e6ceb07170bbea48d266658a2d690c5ce86097fcfd51c3d04fdcb0adb1c3ca9"} Nov 02 20:38:32.055186 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:32.053286 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:32.079232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:32.079202 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-z6cnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 02 20:38:32.079351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:32.079260 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" podUID="27615c4a-c472-4862-b72d-95dff20b301f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 02 20:38:32.111778 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:32.111754 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vxc8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:38:32.111902 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:32.111815 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" podUID="35418b46-29f4-48ba-a0ac-8ca3243c4152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 20:38:33.212694 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.212664 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" Nov 02 20:38:33.302027 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.301992 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/508e995b-a379-439b-a5a1-92e2d46214f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/508e995b-a379-439b-a5a1-92e2d46214f8-maas-api-xks-ca\") pod \"508e995b-a379-439b-a5a1-92e2d46214f8\" (UID: \"508e995b-a379-439b-a5a1-92e2d46214f8\") " Nov 02 20:38:33.302274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.302191 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508e995b-a379-439b-a5a1-92e2d46214f8-kube-api-access-j7vwl\" (UniqueName: \"kubernetes.io/projected/508e995b-a379-439b-a5a1-92e2d46214f8-kube-api-access-j7vwl\") pod \"508e995b-a379-439b-a5a1-92e2d46214f8\" (UID: \"508e995b-a379-439b-a5a1-92e2d46214f8\") " Nov 02 20:38:33.302379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.302312 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/508e995b-a379-439b-a5a1-92e2d46214f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/508e995b-a379-439b-a5a1-92e2d46214f8-maas-api-openshift-ca\") pod \"508e995b-a379-439b-a5a1-92e2d46214f8\" (UID: \"508e995b-a379-439b-a5a1-92e2d46214f8\") " Nov 02 20:38:33.302967 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.302937 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508e995b-a379-439b-a5a1-92e2d46214f8-maas-api-openshift-ca" pod "508e995b-a379-439b-a5a1-92e2d46214f8" (UID: "508e995b-a379-439b-a5a1-92e2d46214f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:38:33.303180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.303158 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/508e995b-a379-439b-a5a1-92e2d46214f8-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:33.306558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.306532 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/508e995b-a379-439b-a5a1-92e2d46214f8-maas-api-xks-ca" pod "508e995b-a379-439b-a5a1-92e2d46214f8" (UID: "508e995b-a379-439b-a5a1-92e2d46214f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:38:33.306674 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.306549 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508e995b-a379-439b-a5a1-92e2d46214f8-kube-api-access-j7vwl" pod "508e995b-a379-439b-a5a1-92e2d46214f8" (UID: "508e995b-a379-439b-a5a1-92e2d46214f8"). InnerVolumeSpecName "kube-api-access-j7vwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:33.420937 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.420799 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/508e995b-a379-439b-a5a1-92e2d46214f8-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:33.420937 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.420840 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7vwl\" (UniqueName: \"kubernetes.io/projected/508e995b-a379-439b-a5a1-92e2d46214f8-kube-api-access-j7vwl\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:33.515036 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:38:33.514989 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:38:33.688780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.688695 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:33.689232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.689209 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:33.690356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.690331 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:38:33.703702 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.703676 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:33.703974 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.703957 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:33.705070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.705048 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:38:33.872054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.872020 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" Nov 02 20:38:33.872054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.872037 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q" event={"ID":"508e995b-a379-439b-a5a1-92e2d46214f8","Type":"ContainerDied","Data":"e486b741a4de83b8f04fc0f2639176eda3d533f2d14bf37392a784dba406e947"} Nov 02 20:38:33.872332 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:33.872122 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e486b741a4de83b8f04fc0f2639176eda3d533f2d14bf37392a784dba406e947" Nov 02 20:38:35.384107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:35.383902 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:36.812668 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:36.812578 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:36.918114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:36.915260 4971 scope.go:122] "RemoveContainer" containerID="c49817ec72189b8d1a44ed95baecf4a57a8082c59125c41361ab5a28a2cd8de9" Nov 02 20:38:37.099335 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:37.066274 4971 scope.go:122] "RemoveContainer" containerID="098172e609de205aeed162d6576008e027cf36281ea4810cee3411a1e4307cee" Nov 02 20:38:37.197141 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:37.178165 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:37.245159 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:37.244504 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:37.342110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:37.280192 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:37.342110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:37.281028 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:37.342110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:37.281761 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 02 20:38:37.360163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:37.345175 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:37.360163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:37.345585 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 20:38:37.509352 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:37.509319 4971 scope.go:122] "RemoveContainer" containerID="5f46c0a8a975accbadefd043f2000b192179c4aabf1b1290412970f5ccb7aa5e" Nov 02 20:38:37.526041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:37.526001 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:37.532199 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:37.532162 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:37.532636 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:37.532607 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 20:38:37.560290 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:37.560262 4971 scope.go:122] "RemoveContainer" containerID="c5638dbccc939ba069356f1fb328360089e4864d4c8f10fba549e4b6614cf523" Nov 02 20:38:37.653166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:37.651230 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 02 20:38:37.693807 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:37.693779 4971 scope.go:122] "RemoveContainer" containerID="1deefaaa9b841d87304dc627a41d9249a8450f7443ac1ac48f87d46d95a7496a" Nov 02 20:38:37.906160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:37.893329 4971 scope.go:122] "RemoveContainer" containerID="b2ee1a5b67dcce77bda867659c3fb57b3177894b4961392b92539f1ef7585335" Nov 02 20:38:38.075107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.070632 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:38.075107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.071249 4971 scope.go:122] "RemoveContainer" containerID="1220ad89a97e422695ce494803b299c84d54331b7e3d25e1159cc2f5595d6e9f" Nov 02 20:38:38.182340 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.182139 4971 scope.go:122] "RemoveContainer" containerID="be25fbc007273fbebb1fc46f992e00cb5f3e6b199ddb724305dd012749b4a1c4" Nov 02 20:38:38.186257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.183704 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:38.197177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.189525 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:38:38.259108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.253400 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:38.259108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.254072 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:38:38.284107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.282779 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:38.284107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.283977 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 20:38:38.284358 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.284309 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 02 20:38:38.291209 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.291183 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:38:38.291454 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.291440 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:38:38.291594 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.291581 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:38:38.351940 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.351804 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:38.353023 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.352888 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:38.422114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.355313 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:38:38.422114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.355373 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:38:38.422114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.381343 4971 scope.go:122] "RemoveContainer" containerID="42913c6414652902ab20ab5023e1a07a9c88ffb52089071d604571fe3a5f32a2" Nov 02 20:38:38.422114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.409786 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Nov 02 20:38:38.467641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.467293 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded" Nov 02 20:38:38.572125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.542212 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:38.572125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.542614 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 02 20:38:38.572125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.542907 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:38.572125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.543241 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:38:38.572125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.543302 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:38:38.572125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.543338 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:38:38.668107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.667543 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:38.668107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:38.667995 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:38:39.207111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.185588 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:39.212927 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.212900 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:39.265115 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.261258 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:39.304760 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.302932 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:39.304928 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.304831 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 20:38:39.321126 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.321064 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 02 20:38:39.366329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.366292 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:39.394327 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.394271 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:39.559517 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.559486 4971 scope.go:122] "RemoveContainer" containerID="cfa8a89d1189486d45c644b4afcb0ae7ee4fde24dfc96a727183a1844d9ef7b7" Nov 02 20:38:39.592366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.561352 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Nov 02 20:38:39.592366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.568635 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn"] Nov 02 20:38:39.592366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.569848 4971 cpu_manager.go:414] "containerMap: removing container" podUID="508e995b-a379-439b-a5a1-92e2d46214f8" containerName="revoke-keys" Nov 02 20:38:39.592366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.569867 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508e995b-a379-439b-a5a1-92e2d46214f8" containerName="revoke-keys" Nov 02 20:38:39.592366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.570128 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="508e995b-a379-439b-a5a1-92e2d46214f8" containerName="revoke-keys" Nov 02 20:38:39.593621 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.593342 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 02 20:38:39.604757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.604170 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:39.683626 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.669384 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:39.857780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.844190 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:38:39.857780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.844365 4971 scope.go:122] "RemoveContainer" containerID="ebbe18bdd7db93087666f82f1d0029349f6e5ab70cd262fa52cf25553b690850" Nov 02 20:38:39.962362 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:39.913175 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:40.026107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.002188 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.051221 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.051182 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.057275 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.057242 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b58ce9c5-f188-4266-98df-0dcee4207d61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.090646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.089185 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.090646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.089262 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.090646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.089348 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.090646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.089417 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.090646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.089456 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv9bs\" (UniqueName: \"kubernetes.io/projected/b58ce9c5-f188-4266-98df-0dcee4207d61-kube-api-access-wv9bs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.090646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.089523 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.239144 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.221815 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:38:40.273423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.260977 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" Nov 02 20:38:40.273423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.261035 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:38:40.273423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.261057 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn"] Nov 02 20:38:40.273423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.226599 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.273423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.262121 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.273423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.262243 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b58ce9c5-f188-4266-98df-0dcee4207d61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.273423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.262333 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.273423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.262387 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.273423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.262460 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.273423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.262529 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.273423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.262568 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv9bs\" (UniqueName: \"kubernetes.io/projected/b58ce9c5-f188-4266-98df-0dcee4207d61-kube-api-access-wv9bs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.273423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.262632 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.307971 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.307929 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 02 20:38:40.329120 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.310996 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b58ce9c5-f188-4266-98df-0dcee4207d61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.462104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.372033 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:40.462104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.226999 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.462104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.388900 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:38:40.462104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.460294 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:38:40.462104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.391174 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.462104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.235188 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:40.470099 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.394871 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.471037 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.396573 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.471971 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.391422 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.483267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.474669 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:38:40.507802 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.507773 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv9bs\" (UniqueName: \"kubernetes.io/projected/b58ce9c5-f188-4266-98df-0dcee4207d61-kube-api-access-wv9bs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.681847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.681537 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.687921 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.687876 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:40.695730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.695693 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:38:40.700109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.697768 4971 scope.go:122] "RemoveContainer" containerID="134b3317fb144f62b37cab54c461c5efa0fa34ebc0e5de3f41f9feb6416b2472" Nov 02 20:38:40.704199 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.704173 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-l2grn\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:40.704492 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.704473 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:38:40.709673 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.709647 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:38:40.768437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.768397 4971 scope.go:122] "RemoveContainer" containerID="2ad3fcf831cfa6d9b204bc17cf21437033d1ae63b5c4e0f17ffdea8b43ec21b3" Nov 02 20:38:40.770677 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.770652 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:38:40.785369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.785342 4971 scope.go:122] "RemoveContainer" containerID="7b6107a0f0d2fb05050ee3ddf351c30022fdcd28fa45f95277098109c90782aa" Nov 02 20:38:40.796698 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.796569 4971 scope.go:122] "RemoveContainer" containerID="f176053d6f6b245ffc5bddecdbf489b8ce7ed444b2de66713c88c6b8bc17d71a" Nov 02 20:38:40.808440 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.808415 4971 scope.go:122] "RemoveContainer" containerID="a76c647af93e9d072405ebf61e41e667661f4225dc18035c6349e79db58f41a2" Nov 02 20:38:40.967457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:40.967425 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:41.482701 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:38:41.482669 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58ce9c5_f188_4266_98df_0dcee4207d61.slice/crio-2b613eb42200485ae6083b75e5319e25b5e4bf42094d0bac858b4874bd24a4f7 WatchSource:0}: Error finding container 2b613eb42200485ae6083b75e5319e25b5e4bf42094d0bac858b4874bd24a4f7: Status 404 returned error can't find the container with id 2b613eb42200485ae6083b75e5319e25b5e4bf42094d0bac858b4874bd24a4f7 Nov 02 20:38:41.489263 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:41.489228 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:41.489368 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:41.489317 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:41.489439 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:41.489388 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:41.493212 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:41.493189 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn"] Nov 02 20:38:42.458602 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:42.458566 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" event={"ID":"b58ce9c5-f188-4266-98df-0dcee4207d61","Type":"ContainerStarted","Data":"b3702c56f0c07814eebe8691d43246d5de11227c2adb4012d41c65f8786e4209"} Nov 02 20:38:42.458790 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:42.458633 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" event={"ID":"b58ce9c5-f188-4266-98df-0dcee4207d61","Type":"ContainerStarted","Data":"2b613eb42200485ae6083b75e5319e25b5e4bf42094d0bac858b4874bd24a4f7"} Nov 02 20:38:42.623107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:42.620731 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:42.635104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:42.630609 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" podStartSLOduration=4.630589585 podStartE2EDuration="4.630589585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:42.622431168 +0000 UTC m=+1450.912087697" watchObservedRunningTime="2026-11-02 20:38:42.630589585 +0000 UTC m=+1450.920246116" Nov 02 20:38:42.879444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:42.879399 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:42.968116 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:42.968068 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:42.969346 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:42.969321 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-l2grn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 02 20:38:42.969476 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:42.969370 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" podUID="b58ce9c5-f188-4266-98df-0dcee4207d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 02 20:38:43.140205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:43.139735 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:43.968758 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:43.968721 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-l2grn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 02 20:38:43.969163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:43.968791 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" podUID="b58ce9c5-f188-4266-98df-0dcee4207d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 02 20:38:44.968914 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:44.968877 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-l2grn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 02 20:38:44.969330 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:44.968957 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" podUID="b58ce9c5-f188-4266-98df-0dcee4207d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 02 20:38:45.969214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:45.968747 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-l2grn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 02 20:38:45.969214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:45.968811 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" podUID="b58ce9c5-f188-4266-98df-0dcee4207d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 02 20:38:46.969043 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:46.968696 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-l2grn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 02 20:38:46.969043 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:46.968746 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" podUID="b58ce9c5-f188-4266-98df-0dcee4207d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 02 20:38:47.810579 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:47.810542 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:47.968059 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:47.968020 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-l2grn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 02 20:38:47.968290 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:47.968134 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" podUID="b58ce9c5-f188-4266-98df-0dcee4207d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 02 20:38:48.968932 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:48.968902 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-l2grn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 02 20:38:48.969335 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:48.968960 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" podUID="b58ce9c5-f188-4266-98df-0dcee4207d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 02 20:38:49.375060 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:49.375022 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:49.585392 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:49.585360 4971 generic.go:356] "Generic (PLEG): container finished" podID="5847fbc4-1639-4ed1-ae14-182e5ab8b50f" containerID="f7b0bbcf8e2d925808102f49126643080e8a7e80ed70ad3c4cd2b23c1a27a3dc" exitCode=0 Nov 02 20:38:49.585560 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:49.585442 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" event={"ID":"5847fbc4-1639-4ed1-ae14-182e5ab8b50f","Type":"ContainerDied","Data":"f7b0bbcf8e2d925808102f49126643080e8a7e80ed70ad3c4cd2b23c1a27a3dc"} Nov 02 20:38:49.586076 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:49.586056 4971 scope.go:122] "RemoveContainer" containerID="f7b0bbcf8e2d925808102f49126643080e8a7e80ed70ad3c4cd2b23c1a27a3dc" Nov 02 20:38:49.588048 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:49.588019 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:49.588150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:49.588118 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:49.588195 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:49.588162 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:49.712195 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:49.712161 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:49.968491 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:49.968406 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-l2grn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 02 20:38:49.968491 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:49.968472 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" podUID="b58ce9c5-f188-4266-98df-0dcee4207d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 02 20:38:50.602931 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:50.602072 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" event={"ID":"5847fbc4-1639-4ed1-ae14-182e5ab8b50f","Type":"ContainerStarted","Data":"e936431e0c6b2176f656474dad00a785e6909bde223516a741fd949af20d4107"} Nov 02 20:38:51.289151 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:51.289123 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:38:51.289404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:51.289379 4971 patch_prober.go:29] interesting pod/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 20:38:51.289460 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:51.289441 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" podUID="5847fbc4-1639-4ed1-ae14-182e5ab8b50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 20:38:51.503246 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:51.503219 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:51.503703 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:51.503684 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:51.504516 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:51.504494 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:38:52.288996 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:52.288963 4971 patch_prober.go:29] interesting pod/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 20:38:52.289512 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:52.289021 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" podUID="5847fbc4-1639-4ed1-ae14-182e5ab8b50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 20:38:52.691506 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:52.691470 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:53.289881 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.289847 4971 patch_prober.go:29] interesting pod/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 20:38:53.290411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.289908 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" podUID="5847fbc4-1639-4ed1-ae14-182e5ab8b50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 20:38:53.413213 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.413181 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf"] Nov 02 20:38:53.417309 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.417289 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.512975 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.512940 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf"] Nov 02 20:38:53.605585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.605545 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.605757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.605592 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.605757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.605710 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.605840 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.605764 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.605918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.605898 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb5da3b-6414-452c-b701-4809b08b5c37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.605955 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.605937 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.606059 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.606048 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjbrq\" (UniqueName: \"kubernetes.io/projected/2bb5da3b-6414-452c-b701-4809b08b5c37-kube-api-access-sjbrq\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.606120 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.606108 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.606197 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.606174 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.707812 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.707781 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb5da3b-6414-452c-b701-4809b08b5c37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.708032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.707823 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.708032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.707882 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjbrq\" (UniqueName: \"kubernetes.io/projected/2bb5da3b-6414-452c-b701-4809b08b5c37-kube-api-access-sjbrq\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.708032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.707921 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.708032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.707958 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.708032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.708028 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.708434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.708048 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.708434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.708132 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.708434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.708185 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.708613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.708589 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb5da3b-6414-452c-b701-4809b08b5c37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.708687 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.708647 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.708752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.708659 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.708808 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.708759 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.709033 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.709012 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.710632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.710609 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.710729 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.710710 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.777411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.777378 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjbrq\" (UniqueName: \"kubernetes.io/projected/2bb5da3b-6414-452c-b701-4809b08b5c37-kube-api-access-sjbrq\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:53.777881 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:53.777849 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-df4jf\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:54.057059 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:54.056975 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:54.289825 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:54.289791 4971 patch_prober.go:29] interesting pod/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 20:38:54.290006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:54.289839 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" podUID="5847fbc4-1639-4ed1-ae14-182e5ab8b50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 20:38:54.486482 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:54.486446 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf"] Nov 02 20:38:54.515290 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:38:54.515260 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb5da3b_6414_452c_b701_4809b08b5c37.slice/crio-543748d87dea7070dcba8139650cde55737aaa4825ee59eac1cde965ff555ade WatchSource:0}: Error finding container 543748d87dea7070dcba8139650cde55737aaa4825ee59eac1cde965ff555ade: Status 404 returned error can't find the container with id 543748d87dea7070dcba8139650cde55737aaa4825ee59eac1cde965ff555ade Nov 02 20:38:54.518125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:54.518049 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:54.518230 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:54.518158 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:54.518230 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:54.518202 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:38:54.665912 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:54.665878 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" event={"ID":"2bb5da3b-6414-452c-b701-4809b08b5c37","Type":"ContainerStarted","Data":"1b7c685c0628138bed89bcb7d280b88f042dd45642a89d50b1dfcaf91b1a01bf"} Nov 02 20:38:54.666125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:54.665936 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" event={"ID":"2bb5da3b-6414-452c-b701-4809b08b5c37","Type":"ContainerStarted","Data":"543748d87dea7070dcba8139650cde55737aaa4825ee59eac1cde965ff555ade"} Nov 02 20:38:54.782203 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:54.782073 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" podStartSLOduration=1.782058307 podStartE2EDuration="1.782058307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:54.780059661 +0000 UTC m=+1463.069716193" watchObservedRunningTime="2026-11-02 20:38:54.782058307 +0000 UTC m=+1463.071714835" Nov 02 20:38:55.057588 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:55.057513 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:38:55.058997 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:55.058975 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-df4jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 02 20:38:55.059121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:55.059019 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" podUID="2bb5da3b-6414-452c-b701-4809b08b5c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 02 20:38:55.289701 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:55.289656 4971 patch_prober.go:29] interesting pod/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 02 20:38:55.289881 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:55.289728 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" podUID="5847fbc4-1639-4ed1-ae14-182e5ab8b50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 02 20:38:56.058264 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:56.058233 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-df4jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 02 20:38:56.058695 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:56.058284 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" podUID="2bb5da3b-6414-452c-b701-4809b08b5c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 02 20:38:56.293570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:56.293541 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:38:56.293962 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:56.293927 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:38:56.294572 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:56.294556 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:38:57.057465 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:57.057426 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-df4jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 02 20:38:57.057681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:57.057487 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" podUID="2bb5da3b-6414-452c-b701-4809b08b5c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 02 20:38:57.520406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:57.520373 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:38:58.057320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:58.057283 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-df4jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 02 20:38:58.057493 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:58.057337 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" podUID="2bb5da3b-6414-452c-b701-4809b08b5c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 02 20:38:59.057753 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:59.057715 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-df4jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 02 20:38:59.058227 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:38:59.057773 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" podUID="2bb5da3b-6414-452c-b701-4809b08b5c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 02 20:39:00.058288 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:00.058256 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-df4jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 02 20:39:00.058688 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:00.058307 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" podUID="2bb5da3b-6414-452c-b701-4809b08b5c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 02 20:39:01.058383 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:01.058338 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-df4jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 02 20:39:01.058769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:01.058409 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" podUID="2bb5da3b-6414-452c-b701-4809b08b5c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 02 20:39:02.057343 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:02.057311 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-df4jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 02 20:39:02.057563 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:02.057355 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" podUID="2bb5da3b-6414-452c-b701-4809b08b5c37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 02 20:39:02.307941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:02.307643 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:03.183687 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:03.183652 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:39:03.184281 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:03.184254 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:39:03.185131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:03.185107 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:39:03.340053 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:39:03.340019 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:39:07.059443 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:07.059397 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:12.074160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:12.073116 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:17.119884 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:17.119843 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:21.971095 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:21.971045 4971 generic.go:356] "Generic (PLEG): container finished" podID="fee3004b-ec0b-4c68-bcab-23b727eb7838" containerID="8ba9cee98bed6ceb1b84bc49b0dd24f01f1ea01fb5cb42b0135ab93aebc54641" exitCode=0 Nov 02 20:39:21.971518 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:21.971107 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" event={"ID":"fee3004b-ec0b-4c68-bcab-23b727eb7838","Type":"ContainerDied","Data":"8ba9cee98bed6ceb1b84bc49b0dd24f01f1ea01fb5cb42b0135ab93aebc54641"} Nov 02 20:39:21.971518 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:21.971504 4971 scope.go:122] "RemoveContainer" containerID="8ba9cee98bed6ceb1b84bc49b0dd24f01f1ea01fb5cb42b0135ab93aebc54641" Nov 02 20:39:21.973122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:21.973092 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:39:21.973184 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:21.973163 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:39:21.973220 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:21.973197 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:39:22.332817 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:22.332689 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:22.982735 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:22.982702 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" event={"ID":"fee3004b-ec0b-4c68-bcab-23b727eb7838","Type":"ContainerStarted","Data":"a301638ef3cc6d42ebdd0ad9f6c7e555a221d6b755a23c2dbc84b692a6f3680e"} Nov 02 20:39:23.221707 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:23.221672 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:39:23.420973 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:23.420941 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:39:23.441255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:23.441216 4971 patch_prober.go:29] interesting pod/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 20:39:23.443545 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:23.443508 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" podUID="fee3004b-ec0b-4c68-bcab-23b727eb7838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 20:39:24.420557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:24.420526 4971 patch_prober.go:29] interesting pod/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 20:39:24.420952 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:24.420579 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" podUID="fee3004b-ec0b-4c68-bcab-23b727eb7838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 20:39:25.421414 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:25.421379 4971 patch_prober.go:29] interesting pod/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 20:39:25.421811 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:25.421439 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" podUID="fee3004b-ec0b-4c68-bcab-23b727eb7838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 20:39:26.420697 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:26.420665 4971 patch_prober.go:29] interesting pod/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 20:39:26.420882 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:26.420720 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" podUID="fee3004b-ec0b-4c68-bcab-23b727eb7838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 20:39:27.162523 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:27.162476 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:27.420801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:27.420709 4971 patch_prober.go:29] interesting pod/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 20:39:27.420801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:27.420779 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" podUID="fee3004b-ec0b-4c68-bcab-23b727eb7838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 20:39:28.215665 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.215627 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt"] Nov 02 20:39:28.220012 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.219991 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt" Nov 02 20:39:28.273951 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.273922 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt"] Nov 02 20:39:28.313530 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.313501 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:39:28.313684 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.313564 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:39:28.350754 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.350719 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d023107-376c-4654-985b-433f2660fff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f1be682-gqrrt\" (UID: \"2d023107-376c-4654-985b-433f2660fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt" Nov 02 20:39:28.351019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.350804 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ztlf\" (UniqueName: \"kubernetes.io/projected/2d023107-376c-4654-985b-433f2660fff2-kube-api-access-7ztlf\") pod \"maas-api-revoke-sub-1f1be682-gqrrt\" (UID: \"2d023107-376c-4654-985b-433f2660fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt" Nov 02 20:39:28.351215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.351145 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d023107-376c-4654-985b-433f2660fff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f1be682-gqrrt\" (UID: \"2d023107-376c-4654-985b-433f2660fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt" Nov 02 20:39:28.421346 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.421320 4971 patch_prober.go:29] interesting pod/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 20:39:28.421503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.421373 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" podUID="fee3004b-ec0b-4c68-bcab-23b727eb7838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 20:39:28.453327 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.453297 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d023107-376c-4654-985b-433f2660fff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f1be682-gqrrt\" (UID: \"2d023107-376c-4654-985b-433f2660fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt" Nov 02 20:39:28.453486 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.453396 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d023107-376c-4654-985b-433f2660fff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f1be682-gqrrt\" (UID: \"2d023107-376c-4654-985b-433f2660fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt" Nov 02 20:39:28.453486 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.453462 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ztlf\" (UniqueName: \"kubernetes.io/projected/2d023107-376c-4654-985b-433f2660fff2-kube-api-access-7ztlf\") pod \"maas-api-revoke-sub-1f1be682-gqrrt\" (UID: \"2d023107-376c-4654-985b-433f2660fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt" Nov 02 20:39:28.454126 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.454099 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d023107-376c-4654-985b-433f2660fff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f1be682-gqrrt\" (UID: \"2d023107-376c-4654-985b-433f2660fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt" Nov 02 20:39:28.456143 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.456071 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d023107-376c-4654-985b-433f2660fff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f1be682-gqrrt\" (UID: \"2d023107-376c-4654-985b-433f2660fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt" Nov 02 20:39:28.511931 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.511875 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ztlf\" (UniqueName: \"kubernetes.io/projected/2d023107-376c-4654-985b-433f2660fff2-kube-api-access-7ztlf\") pod \"maas-api-revoke-sub-1f1be682-gqrrt\" (UID: \"2d023107-376c-4654-985b-433f2660fff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt" Nov 02 20:39:28.557407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.557376 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt" Nov 02 20:39:28.939115 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:39:28.931043 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d023107_376c_4654_985b_433f2660fff2.slice/crio-9ea6f5fb26f051f75570bf999746aa72e49ee470ca68f563dffb2544a1344124 WatchSource:0}: Error finding container 9ea6f5fb26f051f75570bf999746aa72e49ee470ca68f563dffb2544a1344124: Status 404 returned error can't find the container with id 9ea6f5fb26f051f75570bf999746aa72e49ee470ca68f563dffb2544a1344124 Nov 02 20:39:28.939115 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:28.936820 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt"] Nov 02 20:39:29.064371 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:29.064339 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt" event={"ID":"2d023107-376c-4654-985b-433f2660fff2","Type":"ContainerStarted","Data":"9ea6f5fb26f051f75570bf999746aa72e49ee470ca68f563dffb2544a1344124"} Nov 02 20:39:29.425310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:29.425283 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:39:29.426367 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:29.426346 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:39:30.074650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:30.074569 4971 generic.go:356] "Generic (PLEG): container finished" podID="2d023107-376c-4654-985b-433f2660fff2" containerID="13d64a689beb6f8e7442e7192443217c4dae70ec64a3f2499d2482808181ede1" exitCode=0 Nov 02 20:39:30.074650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:30.074614 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt" event={"ID":"2d023107-376c-4654-985b-433f2660fff2","Type":"ContainerDied","Data":"13d64a689beb6f8e7442e7192443217c4dae70ec64a3f2499d2482808181ede1"} Nov 02 20:39:30.174301 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:30.174271 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:30.430352 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:30.430322 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:31.286733 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:31.286709 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt" Nov 02 20:39:31.398471 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:31.398445 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d023107-376c-4654-985b-433f2660fff2-kube-api-access-7ztlf\" (UniqueName: \"kubernetes.io/projected/2d023107-376c-4654-985b-433f2660fff2-kube-api-access-7ztlf\") pod \"2d023107-376c-4654-985b-433f2660fff2\" (UID: \"2d023107-376c-4654-985b-433f2660fff2\") " Nov 02 20:39:31.398643 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:31.398551 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d023107-376c-4654-985b-433f2660fff2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d023107-376c-4654-985b-433f2660fff2-maas-api-xks-ca\") pod \"2d023107-376c-4654-985b-433f2660fff2\" (UID: \"2d023107-376c-4654-985b-433f2660fff2\") " Nov 02 20:39:31.398643 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:31.398588 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d023107-376c-4654-985b-433f2660fff2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d023107-376c-4654-985b-433f2660fff2-maas-api-openshift-ca\") pod \"2d023107-376c-4654-985b-433f2660fff2\" (UID: \"2d023107-376c-4654-985b-433f2660fff2\") " Nov 02 20:39:31.399575 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:31.399551 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d023107-376c-4654-985b-433f2660fff2-maas-api-openshift-ca" pod "2d023107-376c-4654-985b-433f2660fff2" (UID: "2d023107-376c-4654-985b-433f2660fff2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:31.402476 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:31.402454 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d023107-376c-4654-985b-433f2660fff2-maas-api-xks-ca" pod "2d023107-376c-4654-985b-433f2660fff2" (UID: "2d023107-376c-4654-985b-433f2660fff2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:39:31.402689 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:31.402672 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d023107-376c-4654-985b-433f2660fff2-kube-api-access-7ztlf" pod "2d023107-376c-4654-985b-433f2660fff2" (UID: "2d023107-376c-4654-985b-433f2660fff2"). InnerVolumeSpecName "kube-api-access-7ztlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:31.500654 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:31.500619 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d023107-376c-4654-985b-433f2660fff2-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:31.500654 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:31.500649 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d023107-376c-4654-985b-433f2660fff2-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:31.501046 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:31.500663 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ztlf\" (UniqueName: \"kubernetes.io/projected/2d023107-376c-4654-985b-433f2660fff2-kube-api-access-7ztlf\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:31.849936 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:31.849901 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:32.048350 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.048319 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp"] Nov 02 20:39:32.050306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.050272 4971 cpu_manager.go:414] "containerMap: removing container" podUID="2d023107-376c-4654-985b-433f2660fff2" containerName="revoke-keys" Nov 02 20:39:32.050306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.050300 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d023107-376c-4654-985b-433f2660fff2" containerName="revoke-keys" Nov 02 20:39:32.050614 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.050596 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d023107-376c-4654-985b-433f2660fff2" containerName="revoke-keys" Nov 02 20:39:32.054878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.054854 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp" Nov 02 20:39:32.094869 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.094836 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt" Nov 02 20:39:32.095004 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.094884 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt" event={"ID":"2d023107-376c-4654-985b-433f2660fff2","Type":"ContainerDied","Data":"9ea6f5fb26f051f75570bf999746aa72e49ee470ca68f563dffb2544a1344124"} Nov 02 20:39:32.095004 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.094933 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ea6f5fb26f051f75570bf999746aa72e49ee470ca68f563dffb2544a1344124" Nov 02 20:39:32.098992 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.098964 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp"] Nov 02 20:39:32.099814 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.099798 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:39:32.100062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.100017 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:39:32.211274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.211247 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485b8a27-a08d-41a7-a1b2-83ed99558f43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp\" (UID: \"485b8a27-a08d-41a7-a1b2-83ed99558f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp" Nov 02 20:39:32.211543 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.211518 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7d46\" (UniqueName: \"kubernetes.io/projected/485b8a27-a08d-41a7-a1b2-83ed99558f43-kube-api-access-s7d46\") pod \"maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp\" (UID: \"485b8a27-a08d-41a7-a1b2-83ed99558f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp" Nov 02 20:39:32.211595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.211561 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485b8a27-a08d-41a7-a1b2-83ed99558f43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp\" (UID: \"485b8a27-a08d-41a7-a1b2-83ed99558f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp" Nov 02 20:39:32.312762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.312734 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7d46\" (UniqueName: \"kubernetes.io/projected/485b8a27-a08d-41a7-a1b2-83ed99558f43-kube-api-access-s7d46\") pod \"maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp\" (UID: \"485b8a27-a08d-41a7-a1b2-83ed99558f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp" Nov 02 20:39:32.312913 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.312778 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485b8a27-a08d-41a7-a1b2-83ed99558f43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp\" (UID: \"485b8a27-a08d-41a7-a1b2-83ed99558f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp" Nov 02 20:39:32.312971 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.312937 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485b8a27-a08d-41a7-a1b2-83ed99558f43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp\" (UID: \"485b8a27-a08d-41a7-a1b2-83ed99558f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp" Nov 02 20:39:32.313568 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.313545 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485b8a27-a08d-41a7-a1b2-83ed99558f43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp\" (UID: \"485b8a27-a08d-41a7-a1b2-83ed99558f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp" Nov 02 20:39:32.315518 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.315493 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485b8a27-a08d-41a7-a1b2-83ed99558f43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp\" (UID: \"485b8a27-a08d-41a7-a1b2-83ed99558f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp" Nov 02 20:39:32.370886 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.370820 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7d46\" (UniqueName: \"kubernetes.io/projected/485b8a27-a08d-41a7-a1b2-83ed99558f43-kube-api-access-s7d46\") pod \"maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp\" (UID: \"485b8a27-a08d-41a7-a1b2-83ed99558f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp" Nov 02 20:39:32.372533 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.372513 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp" Nov 02 20:39:32.747966 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:39:32.747938 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485b8a27_a08d_41a7_a1b2_83ed99558f43.slice/crio-2ae0b661de939d07745002562ebf7d0ebea7743975f8179970d4c1d4ebbdf772 WatchSource:0}: Error finding container 2ae0b661de939d07745002562ebf7d0ebea7743975f8179970d4c1d4ebbdf772: Status 404 returned error can't find the container with id 2ae0b661de939d07745002562ebf7d0ebea7743975f8179970d4c1d4ebbdf772 Nov 02 20:39:32.751626 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:32.751605 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp"] Nov 02 20:39:33.105630 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:33.105600 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp" event={"ID":"485b8a27-a08d-41a7-a1b2-83ed99558f43","Type":"ContainerStarted","Data":"2ae0b661de939d07745002562ebf7d0ebea7743975f8179970d4c1d4ebbdf772"} Nov 02 20:39:33.221290 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:33.221260 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb"] Nov 02 20:39:33.225335 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:33.225316 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb" Nov 02 20:39:33.280423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:33.280394 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb"] Nov 02 20:39:33.346073 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:33.328121 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fcc284b-51a4-416c-80ee-de4408f10f4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb\" (UID: \"1fcc284b-51a4-416c-80ee-de4408f10f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb" Nov 02 20:39:33.346073 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:33.328198 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds79q\" (UniqueName: \"kubernetes.io/projected/1fcc284b-51a4-416c-80ee-de4408f10f4d-kube-api-access-ds79q\") pod \"maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb\" (UID: \"1fcc284b-51a4-416c-80ee-de4408f10f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb" Nov 02 20:39:33.346073 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:33.328483 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fcc284b-51a4-416c-80ee-de4408f10f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb\" (UID: \"1fcc284b-51a4-416c-80ee-de4408f10f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb" Nov 02 20:39:33.353610 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:39:33.353581 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:39:33.430269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:33.430207 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fcc284b-51a4-416c-80ee-de4408f10f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb\" (UID: \"1fcc284b-51a4-416c-80ee-de4408f10f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb" Nov 02 20:39:33.430382 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:33.430370 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fcc284b-51a4-416c-80ee-de4408f10f4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb\" (UID: \"1fcc284b-51a4-416c-80ee-de4408f10f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb" Nov 02 20:39:33.430424 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:33.430416 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds79q\" (UniqueName: \"kubernetes.io/projected/1fcc284b-51a4-416c-80ee-de4408f10f4d-kube-api-access-ds79q\") pod \"maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb\" (UID: \"1fcc284b-51a4-416c-80ee-de4408f10f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb" Nov 02 20:39:33.430838 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:33.430818 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fcc284b-51a4-416c-80ee-de4408f10f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb\" (UID: \"1fcc284b-51a4-416c-80ee-de4408f10f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb" Nov 02 20:39:33.432766 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:33.432743 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fcc284b-51a4-416c-80ee-de4408f10f4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb\" (UID: \"1fcc284b-51a4-416c-80ee-de4408f10f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb" Nov 02 20:39:33.488634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:33.488610 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds79q\" (UniqueName: \"kubernetes.io/projected/1fcc284b-51a4-416c-80ee-de4408f10f4d-kube-api-access-ds79q\") pod \"maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb\" (UID: \"1fcc284b-51a4-416c-80ee-de4408f10f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb" Nov 02 20:39:33.539816 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:33.539795 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb" Nov 02 20:39:33.915348 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:39:33.915301 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcc284b_51a4_416c_80ee_de4408f10f4d.slice/crio-828ce545690a7d8f426d7257e32eb8e8bbad8bb6f2ffe8947bf70c8049179a81 WatchSource:0}: Error finding container 828ce545690a7d8f426d7257e32eb8e8bbad8bb6f2ffe8947bf70c8049179a81: Status 404 returned error can't find the container with id 828ce545690a7d8f426d7257e32eb8e8bbad8bb6f2ffe8947bf70c8049179a81 Nov 02 20:39:33.918128 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:33.918104 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb"] Nov 02 20:39:34.115522 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:34.115489 4971 generic.go:356] "Generic (PLEG): container finished" podID="485b8a27-a08d-41a7-a1b2-83ed99558f43" containerID="56aefc2a2b9d0f33cc4404cf15961ca3160e2a9f2f813e846612c38194a84dc3" exitCode=0 Nov 02 20:39:34.115698 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:34.115574 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp" event={"ID":"485b8a27-a08d-41a7-a1b2-83ed99558f43","Type":"ContainerDied","Data":"56aefc2a2b9d0f33cc4404cf15961ca3160e2a9f2f813e846612c38194a84dc3"} Nov 02 20:39:34.116994 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:34.116970 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb" event={"ID":"1fcc284b-51a4-416c-80ee-de4408f10f4d","Type":"ContainerStarted","Data":"828ce545690a7d8f426d7257e32eb8e8bbad8bb6f2ffe8947bf70c8049179a81"} Nov 02 20:39:35.127817 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:35.127785 4971 generic.go:356] "Generic (PLEG): container finished" podID="1fcc284b-51a4-416c-80ee-de4408f10f4d" containerID="7f19b2f501711fa531d7a440847f47a00e4c1b9d259a480c1318c496b475472b" exitCode=0 Nov 02 20:39:35.128193 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:35.127872 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb" event={"ID":"1fcc284b-51a4-416c-80ee-de4408f10f4d","Type":"ContainerDied","Data":"7f19b2f501711fa531d7a440847f47a00e4c1b9d259a480c1318c496b475472b"} Nov 02 20:39:35.314808 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:35.314785 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp" Nov 02 20:39:35.462307 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:35.462279 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/485b8a27-a08d-41a7-a1b2-83ed99558f43-kube-api-access-s7d46\" (UniqueName: \"kubernetes.io/projected/485b8a27-a08d-41a7-a1b2-83ed99558f43-kube-api-access-s7d46\") pod \"485b8a27-a08d-41a7-a1b2-83ed99558f43\" (UID: \"485b8a27-a08d-41a7-a1b2-83ed99558f43\") " Nov 02 20:39:35.462481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:35.462327 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/485b8a27-a08d-41a7-a1b2-83ed99558f43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485b8a27-a08d-41a7-a1b2-83ed99558f43-maas-api-openshift-ca\") pod \"485b8a27-a08d-41a7-a1b2-83ed99558f43\" (UID: \"485b8a27-a08d-41a7-a1b2-83ed99558f43\") " Nov 02 20:39:35.462481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:35.462346 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/485b8a27-a08d-41a7-a1b2-83ed99558f43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485b8a27-a08d-41a7-a1b2-83ed99558f43-maas-api-xks-ca\") pod \"485b8a27-a08d-41a7-a1b2-83ed99558f43\" (UID: \"485b8a27-a08d-41a7-a1b2-83ed99558f43\") " Nov 02 20:39:35.462884 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:35.462833 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/485b8a27-a08d-41a7-a1b2-83ed99558f43-maas-api-openshift-ca" pod "485b8a27-a08d-41a7-a1b2-83ed99558f43" (UID: "485b8a27-a08d-41a7-a1b2-83ed99558f43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:35.463528 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:35.463498 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485b8a27-a08d-41a7-a1b2-83ed99558f43-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:35.465035 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:35.465005 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485b8a27-a08d-41a7-a1b2-83ed99558f43-kube-api-access-s7d46" pod "485b8a27-a08d-41a7-a1b2-83ed99558f43" (UID: "485b8a27-a08d-41a7-a1b2-83ed99558f43"). InnerVolumeSpecName "kube-api-access-s7d46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:35.465213 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:35.465028 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/485b8a27-a08d-41a7-a1b2-83ed99558f43-maas-api-xks-ca" pod "485b8a27-a08d-41a7-a1b2-83ed99558f43" (UID: "485b8a27-a08d-41a7-a1b2-83ed99558f43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:39:35.565127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:35.565055 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7d46\" (UniqueName: \"kubernetes.io/projected/485b8a27-a08d-41a7-a1b2-83ed99558f43-kube-api-access-s7d46\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:35.565127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:35.565098 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485b8a27-a08d-41a7-a1b2-83ed99558f43-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:36.138789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:36.138716 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp" Nov 02 20:39:36.139196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:36.138714 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp" event={"ID":"485b8a27-a08d-41a7-a1b2-83ed99558f43","Type":"ContainerDied","Data":"2ae0b661de939d07745002562ebf7d0ebea7743975f8179970d4c1d4ebbdf772"} Nov 02 20:39:36.139196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:36.138841 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ae0b661de939d07745002562ebf7d0ebea7743975f8179970d4c1d4ebbdf772" Nov 02 20:39:36.317256 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:36.317231 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb" Nov 02 20:39:36.375532 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:36.375502 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fcc284b-51a4-416c-80ee-de4408f10f4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fcc284b-51a4-416c-80ee-de4408f10f4d-maas-api-openshift-ca\") pod \"1fcc284b-51a4-416c-80ee-de4408f10f4d\" (UID: \"1fcc284b-51a4-416c-80ee-de4408f10f4d\") " Nov 02 20:39:36.375742 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:36.375687 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fcc284b-51a4-416c-80ee-de4408f10f4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fcc284b-51a4-416c-80ee-de4408f10f4d-maas-api-xks-ca\") pod \"1fcc284b-51a4-416c-80ee-de4408f10f4d\" (UID: \"1fcc284b-51a4-416c-80ee-de4408f10f4d\") " Nov 02 20:39:36.375742 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:36.375723 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fcc284b-51a4-416c-80ee-de4408f10f4d-kube-api-access-ds79q\" (UniqueName: \"kubernetes.io/projected/1fcc284b-51a4-416c-80ee-de4408f10f4d-kube-api-access-ds79q\") pod \"1fcc284b-51a4-416c-80ee-de4408f10f4d\" (UID: \"1fcc284b-51a4-416c-80ee-de4408f10f4d\") " Nov 02 20:39:36.375975 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:36.375950 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fcc284b-51a4-416c-80ee-de4408f10f4d-maas-api-openshift-ca" pod "1fcc284b-51a4-416c-80ee-de4408f10f4d" (UID: "1fcc284b-51a4-416c-80ee-de4408f10f4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:36.376645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:36.376615 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fcc284b-51a4-416c-80ee-de4408f10f4d-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:36.378026 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:36.378003 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fcc284b-51a4-416c-80ee-de4408f10f4d-maas-api-xks-ca" pod "1fcc284b-51a4-416c-80ee-de4408f10f4d" (UID: "1fcc284b-51a4-416c-80ee-de4408f10f4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:39:36.378155 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:36.378076 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcc284b-51a4-416c-80ee-de4408f10f4d-kube-api-access-ds79q" pod "1fcc284b-51a4-416c-80ee-de4408f10f4d" (UID: "1fcc284b-51a4-416c-80ee-de4408f10f4d"). InnerVolumeSpecName "kube-api-access-ds79q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:36.478976 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:36.478944 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fcc284b-51a4-416c-80ee-de4408f10f4d-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:36.479149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:36.478980 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds79q\" (UniqueName: \"kubernetes.io/projected/1fcc284b-51a4-416c-80ee-de4408f10f4d-kube-api-access-ds79q\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:36.929004 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:36.928968 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:37.149282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:37.149250 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb" Nov 02 20:39:37.149651 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:37.149286 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb" event={"ID":"1fcc284b-51a4-416c-80ee-de4408f10f4d","Type":"ContainerDied","Data":"828ce545690a7d8f426d7257e32eb8e8bbad8bb6f2ffe8947bf70c8049179a81"} Nov 02 20:39:37.149651 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:37.149334 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="828ce545690a7d8f426d7257e32eb8e8bbad8bb6f2ffe8947bf70c8049179a81" Nov 02 20:39:37.192439 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:37.192375 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:39.095070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:39.095040 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:42.118025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:42.117985 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:43.454963 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.454919 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r"] Nov 02 20:39:43.455712 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.455692 4971 cpu_manager.go:414] "containerMap: removing container" podUID="485b8a27-a08d-41a7-a1b2-83ed99558f43" containerName="revoke-keys" Nov 02 20:39:43.455712 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.455709 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="485b8a27-a08d-41a7-a1b2-83ed99558f43" containerName="revoke-keys" Nov 02 20:39:43.455911 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.455731 4971 cpu_manager.go:414] "containerMap: removing container" podUID="1fcc284b-51a4-416c-80ee-de4408f10f4d" containerName="revoke-keys" Nov 02 20:39:43.455911 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.455736 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fcc284b-51a4-416c-80ee-de4408f10f4d" containerName="revoke-keys" Nov 02 20:39:43.455911 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.455858 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="485b8a27-a08d-41a7-a1b2-83ed99558f43" containerName="revoke-keys" Nov 02 20:39:43.455911 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.455870 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fcc284b-51a4-416c-80ee-de4408f10f4d" containerName="revoke-keys" Nov 02 20:39:43.459521 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.459207 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" Nov 02 20:39:43.486560 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.486535 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nszd\" (UniqueName: \"kubernetes.io/projected/44a39c6b-dff3-4909-b2ef-1126c3ab8281-kube-api-access-5nszd\") pod \"maas-api-revoke-keys-e2e-derive-574357e2-s964r\" (UID: \"44a39c6b-dff3-4909-b2ef-1126c3ab8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" Nov 02 20:39:43.486663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.486652 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44a39c6b-dff3-4909-b2ef-1126c3ab8281-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-574357e2-s964r\" (UID: \"44a39c6b-dff3-4909-b2ef-1126c3ab8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" Nov 02 20:39:43.486743 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.486730 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44a39c6b-dff3-4909-b2ef-1126c3ab8281-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-574357e2-s964r\" (UID: \"44a39c6b-dff3-4909-b2ef-1126c3ab8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" Nov 02 20:39:43.511908 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.511887 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r"] Nov 02 20:39:43.551312 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.551289 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:39:43.551574 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.551562 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:39:43.588438 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.588411 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nszd\" (UniqueName: \"kubernetes.io/projected/44a39c6b-dff3-4909-b2ef-1126c3ab8281-kube-api-access-5nszd\") pod \"maas-api-revoke-keys-e2e-derive-574357e2-s964r\" (UID: \"44a39c6b-dff3-4909-b2ef-1126c3ab8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" Nov 02 20:39:43.588728 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.588707 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44a39c6b-dff3-4909-b2ef-1126c3ab8281-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-574357e2-s964r\" (UID: \"44a39c6b-dff3-4909-b2ef-1126c3ab8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" Nov 02 20:39:43.588915 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.588898 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44a39c6b-dff3-4909-b2ef-1126c3ab8281-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-574357e2-s964r\" (UID: \"44a39c6b-dff3-4909-b2ef-1126c3ab8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" Nov 02 20:39:43.589787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.589736 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44a39c6b-dff3-4909-b2ef-1126c3ab8281-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-574357e2-s964r\" (UID: \"44a39c6b-dff3-4909-b2ef-1126c3ab8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" Nov 02 20:39:43.591419 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.591397 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44a39c6b-dff3-4909-b2ef-1126c3ab8281-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-574357e2-s964r\" (UID: \"44a39c6b-dff3-4909-b2ef-1126c3ab8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" Nov 02 20:39:43.648496 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.648470 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nszd\" (UniqueName: \"kubernetes.io/projected/44a39c6b-dff3-4909-b2ef-1126c3ab8281-kube-api-access-5nszd\") pod \"maas-api-revoke-keys-e2e-derive-574357e2-s964r\" (UID: \"44a39c6b-dff3-4909-b2ef-1126c3ab8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" Nov 02 20:39:43.711318 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.711251 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p"] Nov 02 20:39:43.773210 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.773175 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" Nov 02 20:39:43.876800 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:43.876777 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr"] Nov 02 20:39:44.217822 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:44.217350 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:44.526134 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:44.492981 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:44.614110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:44.605840 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r"] Nov 02 20:39:44.632573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:44.626679 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" event={"ID":"44a39c6b-dff3-4909-b2ef-1126c3ab8281","Type":"ContainerStarted","Data":"7a9709fef1ffcc208f485651739bf665d75646f3d410e4c93d9bbc76fcccb539"} Nov 02 20:39:45.014581 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:45.014545 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:45.716332 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:45.716292 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" event={"ID":"44a39c6b-dff3-4909-b2ef-1126c3ab8281","Type":"ContainerStarted","Data":"96451e895c08c479094171904e566fc410f3fce78494a51567893026356c199f"} Nov 02 20:39:45.861114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:45.857696 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" podStartSLOduration=2.857673256 podStartE2EDuration="2.857673256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:39:45.82935205 +0000 UTC m=+1514.119008579" watchObservedRunningTime="2026-11-02 20:39:45.857673256 +0000 UTC m=+1514.147329787" Nov 02 20:39:46.737114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:46.736651 4971 generic.go:356] "Generic (PLEG): container finished" podID="44a39c6b-dff3-4909-b2ef-1126c3ab8281" containerID="96451e895c08c479094171904e566fc410f3fce78494a51567893026356c199f" exitCode=0 Nov 02 20:39:46.737114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:46.736962 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" event={"ID":"44a39c6b-dff3-4909-b2ef-1126c3ab8281","Type":"ContainerDied","Data":"96451e895c08c479094171904e566fc410f3fce78494a51567893026356c199f"} Nov 02 20:39:46.974633 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:46.974536 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:47.247245 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:47.245822 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:47.510504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:47.509671 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:48.558836 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:48.552658 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" Nov 02 20:39:48.668466 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:48.666739 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44a39c6b-dff3-4909-b2ef-1126c3ab8281-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44a39c6b-dff3-4909-b2ef-1126c3ab8281-maas-api-xks-ca\") pod \"44a39c6b-dff3-4909-b2ef-1126c3ab8281\" (UID: \"44a39c6b-dff3-4909-b2ef-1126c3ab8281\") " Nov 02 20:39:48.668466 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:48.666882 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a39c6b-dff3-4909-b2ef-1126c3ab8281-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44a39c6b-dff3-4909-b2ef-1126c3ab8281-maas-api-openshift-ca\") pod \"44a39c6b-dff3-4909-b2ef-1126c3ab8281\" (UID: \"44a39c6b-dff3-4909-b2ef-1126c3ab8281\") " Nov 02 20:39:48.668466 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:48.667178 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a39c6b-dff3-4909-b2ef-1126c3ab8281-kube-api-access-5nszd\" (UniqueName: \"kubernetes.io/projected/44a39c6b-dff3-4909-b2ef-1126c3ab8281-kube-api-access-5nszd\") pod \"44a39c6b-dff3-4909-b2ef-1126c3ab8281\" (UID: \"44a39c6b-dff3-4909-b2ef-1126c3ab8281\") " Nov 02 20:39:48.682169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:48.669617 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a39c6b-dff3-4909-b2ef-1126c3ab8281-maas-api-openshift-ca" pod "44a39c6b-dff3-4909-b2ef-1126c3ab8281" (UID: "44a39c6b-dff3-4909-b2ef-1126c3ab8281"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:48.682169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:48.679570 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a39c6b-dff3-4909-b2ef-1126c3ab8281-kube-api-access-5nszd" pod "44a39c6b-dff3-4909-b2ef-1126c3ab8281" (UID: "44a39c6b-dff3-4909-b2ef-1126c3ab8281"). InnerVolumeSpecName "kube-api-access-5nszd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:48.682611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:48.682567 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44a39c6b-dff3-4909-b2ef-1126c3ab8281-maas-api-xks-ca" pod "44a39c6b-dff3-4909-b2ef-1126c3ab8281" (UID: "44a39c6b-dff3-4909-b2ef-1126c3ab8281"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:39:48.717522 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:48.714851 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" podUID="9b9eab18-7c27-4e6c-9bf2-bdfab47f0610" containerName="payload-processing" containerID="cri-o://12d523fbc167030310ffcc141cd8b18045cd5e1c4a836452c4ec255184bf14a1" gracePeriod=25 Nov 02 20:39:48.783398 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:48.783361 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nszd\" (UniqueName: \"kubernetes.io/projected/44a39c6b-dff3-4909-b2ef-1126c3ab8281-kube-api-access-5nszd\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:48.825483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:48.825373 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44a39c6b-dff3-4909-b2ef-1126c3ab8281-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:48.826953 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:48.826916 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44a39c6b-dff3-4909-b2ef-1126c3ab8281-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:48.827145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:48.813597 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" event={"ID":"44a39c6b-dff3-4909-b2ef-1126c3ab8281","Type":"ContainerDied","Data":"7a9709fef1ffcc208f485651739bf665d75646f3d410e4c93d9bbc76fcccb539"} Nov 02 20:39:48.837898 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:48.837878 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a9709fef1ffcc208f485651739bf665d75646f3d410e4c93d9bbc76fcccb539" Nov 02 20:39:48.838059 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:48.813736 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r" Nov 02 20:39:48.877186 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:48.877141 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" podUID="e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17" containerName="payload-pre-processing" containerID="cri-o://9837563d65069e057c5ddbd865c7342134a6b0b1be4f107b93c3233b683a5a8e" gracePeriod=25 Nov 02 20:39:49.172976 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.172677 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" Nov 02 20:39:49.233301 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.227688 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b9eab18-7c27-4e6c-9bf2-bdfab47f0610-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9eab18-7c27-4e6c-9bf2-bdfab47f0610-plugins-config-volume\") pod \"9b9eab18-7c27-4e6c-9bf2-bdfab47f0610\" (UID: \"9b9eab18-7c27-4e6c-9bf2-bdfab47f0610\") " Nov 02 20:39:49.233301 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.228065 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9eab18-7c27-4e6c-9bf2-bdfab47f0610-kube-api-access-8wz4q\" (UniqueName: \"kubernetes.io/projected/9b9eab18-7c27-4e6c-9bf2-bdfab47f0610-kube-api-access-8wz4q\") pod \"9b9eab18-7c27-4e6c-9bf2-bdfab47f0610\" (UID: \"9b9eab18-7c27-4e6c-9bf2-bdfab47f0610\") " Nov 02 20:39:49.233301 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.229520 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9eab18-7c27-4e6c-9bf2-bdfab47f0610-plugins-config-volume" pod "9b9eab18-7c27-4e6c-9bf2-bdfab47f0610" (UID: "9b9eab18-7c27-4e6c-9bf2-bdfab47f0610"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:49.249474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.249445 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9eab18-7c27-4e6c-9bf2-bdfab47f0610-kube-api-access-8wz4q" pod "9b9eab18-7c27-4e6c-9bf2-bdfab47f0610" (UID: "9b9eab18-7c27-4e6c-9bf2-bdfab47f0610"). InnerVolumeSpecName "kube-api-access-8wz4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:49.342732 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.333421 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wz4q\" (UniqueName: \"kubernetes.io/projected/9b9eab18-7c27-4e6c-9bf2-bdfab47f0610-kube-api-access-8wz4q\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:49.342732 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.333460 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9eab18-7c27-4e6c-9bf2-bdfab47f0610-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:49.445552 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.445126 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" Nov 02 20:39:49.537215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.537189 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17-kube-api-access-c76q7\" (UniqueName: \"kubernetes.io/projected/e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17-kube-api-access-c76q7\") pod \"e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17\" (UID: \"e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17\") " Nov 02 20:39:49.537378 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.537304 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17-plugins-config-volume\") pod \"e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17\" (UID: \"e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17\") " Nov 02 20:39:49.538011 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.537986 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17-plugins-config-volume" pod "e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17" (UID: "e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:49.540343 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.540311 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17-kube-api-access-c76q7" pod "e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17" (UID: "e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17"). InnerVolumeSpecName "kube-api-access-c76q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:49.651111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.641531 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:49.651111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.641575 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c76q7\" (UniqueName: \"kubernetes.io/projected/e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17-kube-api-access-c76q7\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:49.862165 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.854351 4971 generic.go:356] "Generic (PLEG): container finished" podID="e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17" containerID="9837563d65069e057c5ddbd865c7342134a6b0b1be4f107b93c3233b683a5a8e" exitCode=0 Nov 02 20:39:49.862165 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.856296 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" Nov 02 20:39:49.907109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.904966 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" event={"ID":"e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17","Type":"ContainerDied","Data":"9837563d65069e057c5ddbd865c7342134a6b0b1be4f107b93c3233b683a5a8e"} Nov 02 20:39:49.907109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.905055 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr" event={"ID":"e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17","Type":"ContainerDied","Data":"0cb3cb81effcff437b36d1ac3daafda2fac9a9ba2b0af9af281f11dcd543c96f"} Nov 02 20:39:49.907109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.905110 4971 scope.go:122] "RemoveContainer" containerID="9837563d65069e057c5ddbd865c7342134a6b0b1be4f107b93c3233b683a5a8e" Nov 02 20:39:49.914551 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.913005 4971 generic.go:356] "Generic (PLEG): container finished" podID="9b9eab18-7c27-4e6c-9bf2-bdfab47f0610" containerID="12d523fbc167030310ffcc141cd8b18045cd5e1c4a836452c4ec255184bf14a1" exitCode=0 Nov 02 20:39:49.914551 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.913128 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" Nov 02 20:39:49.945687 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.945657 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" event={"ID":"9b9eab18-7c27-4e6c-9bf2-bdfab47f0610","Type":"ContainerDied","Data":"12d523fbc167030310ffcc141cd8b18045cd5e1c4a836452c4ec255184bf14a1"} Nov 02 20:39:49.945896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:49.945874 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p" event={"ID":"9b9eab18-7c27-4e6c-9bf2-bdfab47f0610","Type":"ContainerDied","Data":"e91f83e4fe9ecf4a3fa71593f394b9b954e508e162ca94aa9330ec039de82fea"} Nov 02 20:39:50.050434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:50.050107 4971 scope.go:122] "RemoveContainer" containerID="9837563d65069e057c5ddbd865c7342134a6b0b1be4f107b93c3233b683a5a8e" Nov 02 20:39:50.050964 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:39:50.050627 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9837563d65069e057c5ddbd865c7342134a6b0b1be4f107b93c3233b683a5a8e\": container with ID starting with 9837563d65069e057c5ddbd865c7342134a6b0b1be4f107b93c3233b683a5a8e not found: ID does not exist" containerID="9837563d65069e057c5ddbd865c7342134a6b0b1be4f107b93c3233b683a5a8e" Nov 02 20:39:50.050964 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:50.050663 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9837563d65069e057c5ddbd865c7342134a6b0b1be4f107b93c3233b683a5a8e"} err="failed to get container status \"9837563d65069e057c5ddbd865c7342134a6b0b1be4f107b93c3233b683a5a8e\": rpc error: code = NotFound desc = could not find container \"9837563d65069e057c5ddbd865c7342134a6b0b1be4f107b93c3233b683a5a8e\": container with ID starting with 9837563d65069e057c5ddbd865c7342134a6b0b1be4f107b93c3233b683a5a8e not found: ID does not exist" Nov 02 20:39:50.050964 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:50.050687 4971 scope.go:122] "RemoveContainer" containerID="12d523fbc167030310ffcc141cd8b18045cd5e1c4a836452c4ec255184bf14a1" Nov 02 20:39:50.072989 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:50.072932 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p"] Nov 02 20:39:50.086246 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:50.086186 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b3e20d8-f97f98dd6-8jl8p"] Nov 02 20:39:50.112437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:50.112065 4971 scope.go:122] "RemoveContainer" containerID="12d523fbc167030310ffcc141cd8b18045cd5e1c4a836452c4ec255184bf14a1" Nov 02 20:39:50.113615 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:39:50.113485 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d523fbc167030310ffcc141cd8b18045cd5e1c4a836452c4ec255184bf14a1\": container with ID starting with 12d523fbc167030310ffcc141cd8b18045cd5e1c4a836452c4ec255184bf14a1 not found: ID does not exist" containerID="12d523fbc167030310ffcc141cd8b18045cd5e1c4a836452c4ec255184bf14a1" Nov 02 20:39:50.113615 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:50.113527 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d523fbc167030310ffcc141cd8b18045cd5e1c4a836452c4ec255184bf14a1"} err="failed to get container status \"12d523fbc167030310ffcc141cd8b18045cd5e1c4a836452c4ec255184bf14a1\": rpc error: code = NotFound desc = could not find container \"12d523fbc167030310ffcc141cd8b18045cd5e1c4a836452c4ec255184bf14a1\": container with ID starting with 12d523fbc167030310ffcc141cd8b18045cd5e1c4a836452c4ec255184bf14a1 not found: ID does not exist" Nov 02 20:39:50.178483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:50.178417 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr"] Nov 02 20:39:50.190943 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:50.190910 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b3e20d8-8564d4494c-x64vr"] Nov 02 20:39:51.878566 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:51.878532 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b9eab18-7c27-4e6c-9bf2-bdfab47f0610" path="/var/lib/kubelet/pods/9b9eab18-7c27-4e6c-9bf2-bdfab47f0610/volumes" Nov 02 20:39:51.879957 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:51.879930 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17" path="/var/lib/kubelet/pods/e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17/volumes" Nov 02 20:39:52.569897 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:52.569855 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:53.744771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:53.744740 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:55.188567 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:55.188534 4971 generic.go:356] "Generic (PLEG): container finished" podID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" containerID="123686016c6f8ee17ad443eb47bce41508ba670fb3e6b680aa1926e8abf5fca7" exitCode=0 Nov 02 20:39:55.189003 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:55.188607 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" event={"ID":"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a","Type":"ContainerDied","Data":"123686016c6f8ee17ad443eb47bce41508ba670fb3e6b680aa1926e8abf5fca7"} Nov 02 20:39:55.189135 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:55.189118 4971 scope.go:122] "RemoveContainer" containerID="123686016c6f8ee17ad443eb47bce41508ba670fb3e6b680aa1926e8abf5fca7" Nov 02 20:39:55.190757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:55.190721 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:39:55.190883 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:55.190811 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:39:55.190883 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:55.190857 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:39:55.569786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:55.569649 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:55.817904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:55.817869 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:56.199476 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:56.199444 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" event={"ID":"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a","Type":"ContainerStarted","Data":"c881f3cc4e3421e08aa6a1a4cc9924e4a948a9e43f140b3c40a3515a27ca9323"} Nov 02 20:39:56.248486 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:56.247704 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:39:56.248834 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:56.248802 4971 patch_prober.go:29] interesting pod/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 20:39:56.248978 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:56.248878 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" podUID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 20:39:57.247235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:57.247197 4971 patch_prober.go:29] interesting pod/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 20:39:57.247663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:57.247255 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" podUID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 20:39:57.483171 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:57.483132 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:39:58.240800 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:58.240757 4971 patch_prober.go:29] interesting pod/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 20:39:58.241209 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:58.241164 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" podUID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 20:39:59.247635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:59.247596 4971 patch_prober.go:29] interesting pod/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 20:39:59.248166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:59.247669 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" podUID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 20:39:59.352301 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:39:59.352252 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:40:00.048281 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:00.048244 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:00.239106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:00.235387 4971 patch_prober.go:29] interesting pod/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 20:40:00.239106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:00.235458 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" podUID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 20:40:01.242110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:01.241456 4971 patch_prober.go:29] interesting pod/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 20:40:01.242110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:01.241521 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" podUID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 20:40:02.238664 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:02.238636 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:40:02.239838 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:02.239809 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:40:02.335008 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:02.334972 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:03.342128 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:40:03.342049 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:40:04.049921 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:04.049894 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp"] Nov 02 20:40:04.096970 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:04.096943 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517db615-n7xzp"] Nov 02 20:40:05.880900 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:05.880864 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38880071-38b8-459e-a7b2-9ab3875087bd" path="/var/lib/kubelet/pods/38880071-38b8-459e-a7b2-9ab3875087bd/volumes" Nov 02 20:40:06.622263 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:06.622227 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:08.178713 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:08.178681 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:08.431337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:08.431236 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:08.681592 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:08.681515 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:08.728860 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:08.728824 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp"] Nov 02 20:40:08.776215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:08.776186 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b3e20d8-4f9lp"] Nov 02 20:40:08.804343 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:08.804312 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9"] Nov 02 20:40:08.804614 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:08.804586 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" podUID="5847fbc4-1639-4ed1-ae14-182e5ab8b50f" containerName="istio-proxy" containerID="cri-o://e936431e0c6b2176f656474dad00a785e6909bde223516a741fd949af20d4107" gracePeriod=30 Nov 02 20:40:09.026268 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:09.023579 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:09.638237 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:09.638202 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:09.869450 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:09.869415 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="485b8a27-a08d-41a7-a1b2-83ed99558f43" path="/var/lib/kubelet/pods/485b8a27-a08d-41a7-a1b2-83ed99558f43/volumes" Nov 02 20:40:10.338376 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.338342 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz"] Nov 02 20:40:10.339131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.339114 4971 cpu_manager.go:414] "containerMap: removing container" podUID="9b9eab18-7c27-4e6c-9bf2-bdfab47f0610" containerName="payload-processing" Nov 02 20:40:10.339131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.339131 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b9eab18-7c27-4e6c-9bf2-bdfab47f0610" containerName="payload-processing" Nov 02 20:40:10.339252 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.339159 4971 cpu_manager.go:414] "containerMap: removing container" podUID="44a39c6b-dff3-4909-b2ef-1126c3ab8281" containerName="revoke-keys" Nov 02 20:40:10.339252 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.339164 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a39c6b-dff3-4909-b2ef-1126c3ab8281" containerName="revoke-keys" Nov 02 20:40:10.339252 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.339190 4971 cpu_manager.go:414] "containerMap: removing container" podUID="e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17" containerName="payload-pre-processing" Nov 02 20:40:10.339252 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.339194 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17" containerName="payload-pre-processing" Nov 02 20:40:10.339380 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.339311 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a39c6b-dff3-4909-b2ef-1126c3ab8281" containerName="revoke-keys" Nov 02 20:40:10.339380 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.339323 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="e26c8fd8-69ac-4f08-8fea-6ad4bfb7dc17" containerName="payload-pre-processing" Nov 02 20:40:10.339380 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.339335 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b9eab18-7c27-4e6c-9bf2-bdfab47f0610" containerName="payload-processing" Nov 02 20:40:10.343571 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.343552 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.401727 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.401697 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-df17143f-openshift-default-dockercfg-stx7h\"" Nov 02 20:40:10.414401 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.414372 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-token\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.414557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.414419 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a50c847f-3e10-42a9-8ca1-1e285db2af51-istiod-ca-cert\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.414557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.414472 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bblnq\" (UniqueName: \"kubernetes.io/projected/a50c847f-3e10-42a9-8ca1-1e285db2af51-kube-api-access-bblnq\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.414557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.414518 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-workload-certs\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.414785 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.414769 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-envoy\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.414856 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.414813 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-credential-socket\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.414915 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.414882 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-podinfo\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.414977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.414924 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-workload-socket\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.415051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.414972 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-data\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.457645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.457502 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz"] Nov 02 20:40:10.518068 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.518036 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-envoy\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.518297 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.518186 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-credential-socket\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.518297 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.518267 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-podinfo\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.518433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.518330 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-workload-socket\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.518433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.518387 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-data\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.518574 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.518451 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-token\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.518574 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.518487 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a50c847f-3e10-42a9-8ca1-1e285db2af51-istiod-ca-cert\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.518574 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.518543 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bblnq\" (UniqueName: \"kubernetes.io/projected/a50c847f-3e10-42a9-8ca1-1e285db2af51-kube-api-access-bblnq\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.518729 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.518595 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-workload-certs\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.519255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.519223 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-workload-socket\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.519428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.519266 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-workload-certs\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.519512 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.519495 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-data\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.520270 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.520241 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a50c847f-3e10-42a9-8ca1-1e285db2af51-istiod-ca-cert\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.520409 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.520325 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-credential-socket\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.525631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.525606 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-podinfo\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.527305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.527264 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-envoy\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.545106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.545051 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:10.580531 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.580497 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bblnq\" (UniqueName: \"kubernetes.io/projected/a50c847f-3e10-42a9-8ca1-1e285db2af51-kube-api-access-bblnq\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.580836 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.580812 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-token\") pod \"e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.658525 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.658496 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:10.796236 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:10.796201 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:11.113962 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:40:11.113934 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50c847f_3e10_42a9_8ca1_1e285db2af51.slice/crio-3f390780834bc4e2003a5137c3aba7ec3de1bb48d604a97508b67fbd73a48b8c WatchSource:0}: Error finding container 3f390780834bc4e2003a5137c3aba7ec3de1bb48d604a97508b67fbd73a48b8c: Status 404 returned error can't find the container with id 3f390780834bc4e2003a5137c3aba7ec3de1bb48d604a97508b67fbd73a48b8c Nov 02 20:40:11.116254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:11.116230 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz"] Nov 02 20:40:11.116603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:11.116570 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:40:11.116648 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:11.116637 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:40:11.116683 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:11.116668 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:40:11.295673 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:11.295633 4971 patch_prober.go:29] interesting pod/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:40:11.295868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:11.295691 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" podUID="5847fbc4-1639-4ed1-ae14-182e5ab8b50f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:40:11.478998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:11.478896 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" event={"ID":"a50c847f-3e10-42a9-8ca1-1e285db2af51","Type":"ContainerStarted","Data":"cd843bc1894ec2b4d1b846cf3045d9c6371356d9a2e5abccf93ffd69a7e6ec46"} Nov 02 20:40:11.478998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:11.478958 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" event={"ID":"a50c847f-3e10-42a9-8ca1-1e285db2af51","Type":"ContainerStarted","Data":"3f390780834bc4e2003a5137c3aba7ec3de1bb48d604a97508b67fbd73a48b8c"} Nov 02 20:40:11.593617 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:11.593566 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" podStartSLOduration=1.593551512 podStartE2EDuration="1.593551512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:40:11.592304202 +0000 UTC m=+1539.881960732" watchObservedRunningTime="2026-11-02 20:40:11.593551512 +0000 UTC m=+1539.883208040" Nov 02 20:40:11.658752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:11.658727 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:12.664678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:12.664648 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:13.509612 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:13.509573 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:13.510378 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:13.510356 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:40:14.062997 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.062966 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:14.108391 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.108365 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:40:14.199646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.199613 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-data\") pod \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " Nov 02 20:40:14.199798 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.199652 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-token\" (UniqueName: \"kubernetes.io/projected/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-token\") pod \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " Nov 02 20:40:14.199798 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.199672 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-kube-api-access-mzh9h\" (UniqueName: \"kubernetes.io/projected/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-kube-api-access-mzh9h\") pod \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " Nov 02 20:40:14.199798 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.199703 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-credential-socket\") pod \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " Nov 02 20:40:14.199972 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.199818 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istiod-ca-cert\") pod \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " Nov 02 20:40:14.199972 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.199843 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-workload-certs\") pod \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " Nov 02 20:40:14.199972 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.199925 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-podinfo\") pod \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " Nov 02 20:40:14.199972 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.199935 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-data" pod "5847fbc4-1639-4ed1-ae14-182e5ab8b50f" (UID: "5847fbc4-1639-4ed1-ae14-182e5ab8b50f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:14.200214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.199984 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-workload-socket\") pod \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " Nov 02 20:40:14.200214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.200058 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-envoy\") pod \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\" (UID: \"5847fbc4-1639-4ed1-ae14-182e5ab8b50f\") " Nov 02 20:40:14.200214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.200122 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-credential-socket" pod "5847fbc4-1639-4ed1-ae14-182e5ab8b50f" (UID: "5847fbc4-1639-4ed1-ae14-182e5ab8b50f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:14.200214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.200198 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-workload-certs" pod "5847fbc4-1639-4ed1-ae14-182e5ab8b50f" (UID: "5847fbc4-1639-4ed1-ae14-182e5ab8b50f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:14.200423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.200316 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istiod-ca-cert" pod "5847fbc4-1639-4ed1-ae14-182e5ab8b50f" (UID: "5847fbc4-1639-4ed1-ae14-182e5ab8b50f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:40:14.200472 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.200415 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-workload-socket" pod "5847fbc4-1639-4ed1-ae14-182e5ab8b50f" (UID: "5847fbc4-1639-4ed1-ae14-182e5ab8b50f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:14.201047 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.201022 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:14.201177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.201051 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:14.201177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.201067 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:14.201177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.201099 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:14.201177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.201115 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:14.202392 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.202368 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-token" pod "5847fbc4-1639-4ed1-ae14-182e5ab8b50f" (UID: "5847fbc4-1639-4ed1-ae14-182e5ab8b50f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:40:14.202493 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.202403 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-kube-api-access-mzh9h" pod "5847fbc4-1639-4ed1-ae14-182e5ab8b50f" (UID: "5847fbc4-1639-4ed1-ae14-182e5ab8b50f"). InnerVolumeSpecName "kube-api-access-mzh9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:40:14.202493 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.202460 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-envoy" pod "5847fbc4-1639-4ed1-ae14-182e5ab8b50f" (UID: "5847fbc4-1639-4ed1-ae14-182e5ab8b50f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:14.202575 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.202536 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-podinfo" pod "5847fbc4-1639-4ed1-ae14-182e5ab8b50f" (UID: "5847fbc4-1639-4ed1-ae14-182e5ab8b50f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:40:14.303346 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.303313 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:14.303477 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.303352 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzh9h\" (UniqueName: \"kubernetes.io/projected/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-kube-api-access-mzh9h\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:14.303477 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.303370 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:14.303477 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.303383 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5847fbc4-1639-4ed1-ae14-182e5ab8b50f-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:14.316771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.316736 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:14.522072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.522033 4971 generic.go:356] "Generic (PLEG): container finished" podID="5847fbc4-1639-4ed1-ae14-182e5ab8b50f" containerID="e936431e0c6b2176f656474dad00a785e6909bde223516a741fd949af20d4107" exitCode=0 Nov 02 20:40:14.522275 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.522112 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" Nov 02 20:40:14.522275 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.522124 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" event={"ID":"5847fbc4-1639-4ed1-ae14-182e5ab8b50f","Type":"ContainerDied","Data":"e936431e0c6b2176f656474dad00a785e6909bde223516a741fd949af20d4107"} Nov 02 20:40:14.522275 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.522189 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9" event={"ID":"5847fbc4-1639-4ed1-ae14-182e5ab8b50f","Type":"ContainerDied","Data":"f1fec747db8c4b2a65b32422da5bd93cfd4894b0f87e1eeb79524e6b11df1f2b"} Nov 02 20:40:14.522275 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.522216 4971 scope.go:122] "RemoveContainer" containerID="e936431e0c6b2176f656474dad00a785e6909bde223516a741fd949af20d4107" Nov 02 20:40:14.534977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.534958 4971 scope.go:122] "RemoveContainer" containerID="f7b0bbcf8e2d925808102f49126643080e8a7e80ed70ad3c4cd2b23c1a27a3dc" Nov 02 20:40:14.546173 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.546156 4971 scope.go:122] "RemoveContainer" containerID="e936431e0c6b2176f656474dad00a785e6909bde223516a741fd949af20d4107" Nov 02 20:40:14.546462 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:40:14.546442 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e936431e0c6b2176f656474dad00a785e6909bde223516a741fd949af20d4107\": container with ID starting with e936431e0c6b2176f656474dad00a785e6909bde223516a741fd949af20d4107 not found: ID does not exist" containerID="e936431e0c6b2176f656474dad00a785e6909bde223516a741fd949af20d4107" Nov 02 20:40:14.546578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.546473 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e936431e0c6b2176f656474dad00a785e6909bde223516a741fd949af20d4107"} err="failed to get container status \"e936431e0c6b2176f656474dad00a785e6909bde223516a741fd949af20d4107\": rpc error: code = NotFound desc = could not find container \"e936431e0c6b2176f656474dad00a785e6909bde223516a741fd949af20d4107\": container with ID starting with e936431e0c6b2176f656474dad00a785e6909bde223516a741fd949af20d4107 not found: ID does not exist" Nov 02 20:40:14.546578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.546491 4971 scope.go:122] "RemoveContainer" containerID="f7b0bbcf8e2d925808102f49126643080e8a7e80ed70ad3c4cd2b23c1a27a3dc" Nov 02 20:40:14.546774 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:40:14.546755 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b0bbcf8e2d925808102f49126643080e8a7e80ed70ad3c4cd2b23c1a27a3dc\": container with ID starting with f7b0bbcf8e2d925808102f49126643080e8a7e80ed70ad3c4cd2b23c1a27a3dc not found: ID does not exist" containerID="f7b0bbcf8e2d925808102f49126643080e8a7e80ed70ad3c4cd2b23c1a27a3dc" Nov 02 20:40:14.546845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.546780 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b0bbcf8e2d925808102f49126643080e8a7e80ed70ad3c4cd2b23c1a27a3dc"} err="failed to get container status \"f7b0bbcf8e2d925808102f49126643080e8a7e80ed70ad3c4cd2b23c1a27a3dc\": rpc error: code = NotFound desc = could not find container \"f7b0bbcf8e2d925808102f49126643080e8a7e80ed70ad3c4cd2b23c1a27a3dc\": container with ID starting with f7b0bbcf8e2d925808102f49126643080e8a7e80ed70ad3c4cd2b23c1a27a3dc not found: ID does not exist" Nov 02 20:40:14.682962 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.682934 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9"] Nov 02 20:40:14.729896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:14.729870 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2b3e20d8-openshift-default-5f79f8f569-5vhx9"] Nov 02 20:40:15.868701 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:15.868667 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5847fbc4-1639-4ed1-ae14-182e5ab8b50f" path="/var/lib/kubelet/pods/5847fbc4-1639-4ed1-ae14-182e5ab8b50f/volumes" Nov 02 20:40:17.087788 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.087696 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:17.358659 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.358600 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:17.519670 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.519631 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb"] Nov 02 20:40:17.521133 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.521069 4971 cpu_manager.go:414] "containerMap: removing container" podUID="5847fbc4-1639-4ed1-ae14-182e5ab8b50f" containerName="istio-proxy" Nov 02 20:40:17.521309 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.521295 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5847fbc4-1639-4ed1-ae14-182e5ab8b50f" containerName="istio-proxy" Nov 02 20:40:17.521699 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.521682 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="5847fbc4-1639-4ed1-ae14-182e5ab8b50f" containerName="istio-proxy" Nov 02 20:40:17.521807 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.521796 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="5847fbc4-1639-4ed1-ae14-182e5ab8b50f" containerName="istio-proxy" Nov 02 20:40:17.549130 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.548402 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" Nov 02 20:40:17.578075 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.578040 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb"] Nov 02 20:40:17.600751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.598318 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-df17143f\"" Nov 02 20:40:17.616775 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.615793 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-df17143f-dockercfg-wnfhk\"" Nov 02 20:40:17.705750 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.702600 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d35c028-a868-46ca-8be5-6cd3023d724c-plugins-config-volume\") pod \"payload-processing-e2e-ait-df17143f-f5d9549d-df4wb\" (UID: \"4d35c028-a868-46ca-8be5-6cd3023d724c\") " pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" Nov 02 20:40:17.705750 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.702923 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qr5l\" (UniqueName: \"kubernetes.io/projected/4d35c028-a868-46ca-8be5-6cd3023d724c-kube-api-access-7qr5l\") pod \"payload-processing-e2e-ait-df17143f-f5d9549d-df4wb\" (UID: \"4d35c028-a868-46ca-8be5-6cd3023d724c\") " pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" Nov 02 20:40:17.805155 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.805109 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d35c028-a868-46ca-8be5-6cd3023d724c-plugins-config-volume\") pod \"payload-processing-e2e-ait-df17143f-f5d9549d-df4wb\" (UID: \"4d35c028-a868-46ca-8be5-6cd3023d724c\") " pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" Nov 02 20:40:17.805347 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.805273 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qr5l\" (UniqueName: \"kubernetes.io/projected/4d35c028-a868-46ca-8be5-6cd3023d724c-kube-api-access-7qr5l\") pod \"payload-processing-e2e-ait-df17143f-f5d9549d-df4wb\" (UID: \"4d35c028-a868-46ca-8be5-6cd3023d724c\") " pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" Nov 02 20:40:17.806157 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.806117 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d35c028-a868-46ca-8be5-6cd3023d724c-plugins-config-volume\") pod \"payload-processing-e2e-ait-df17143f-f5d9549d-df4wb\" (UID: \"4d35c028-a868-46ca-8be5-6cd3023d724c\") " pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" Nov 02 20:40:17.863052 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.863023 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qr5l\" (UniqueName: \"kubernetes.io/projected/4d35c028-a868-46ca-8be5-6cd3023d724c-kube-api-access-7qr5l\") pod \"payload-processing-e2e-ait-df17143f-f5d9549d-df4wb\" (UID: \"4d35c028-a868-46ca-8be5-6cd3023d724c\") " pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" Nov 02 20:40:17.877779 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:17.877697 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" Nov 02 20:40:18.255037 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:40:18.255002 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d35c028_a868_46ca_8be5_6cd3023d724c.slice/crio-07ac26eb55965ff7106b13560e5d79c771c55ac4ed9230a231e8607574428ff5 WatchSource:0}: Error finding container 07ac26eb55965ff7106b13560e5d79c771c55ac4ed9230a231e8607574428ff5: Status 404 returned error can't find the container with id 07ac26eb55965ff7106b13560e5d79c771c55ac4ed9230a231e8607574428ff5 Nov 02 20:40:18.258191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:18.258160 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb"] Nov 02 20:40:18.362144 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:18.360925 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:18.598444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:18.598403 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" event={"ID":"4d35c028-a868-46ca-8be5-6cd3023d724c","Type":"ContainerStarted","Data":"5d7b0ad88249ddf9247eb9fd4f4251fe9f23e1dea075c1d0d0deb05b473686d5"} Nov 02 20:40:18.598646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:18.598471 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" event={"ID":"4d35c028-a868-46ca-8be5-6cd3023d724c","Type":"ContainerStarted","Data":"07ac26eb55965ff7106b13560e5d79c771c55ac4ed9230a231e8607574428ff5"} Nov 02 20:40:18.598646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:18.598497 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" Nov 02 20:40:18.716153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:18.716069 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" podStartSLOduration=1.716050362 podStartE2EDuration="1.716050362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:40:18.709447237 +0000 UTC m=+1546.999103766" watchObservedRunningTime="2026-11-02 20:40:18.716050362 +0000 UTC m=+1547.005706891" Nov 02 20:40:19.987278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:19.987251 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:20.237780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:20.237672 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:20.497199 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:20.497112 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb"] Nov 02 20:40:20.543431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:20.543389 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6c66f658-vfwfb"] Nov 02 20:40:20.607071 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:20.607040 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:20.611441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:20.611414 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7"] Nov 02 20:40:20.611681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:20.611647 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" podUID="fee3004b-ec0b-4c68-bcab-23b727eb7838" containerName="istio-proxy" containerID="cri-o://a301638ef3cc6d42ebdd0ad9f6c7e555a221d6b755a23c2dbc84b692a6f3680e" gracePeriod=30 Nov 02 20:40:20.879104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:20.878139 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:21.350453 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:21.350401 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:21.999949 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:21.999896 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:22.002814 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:22.002783 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fcc284b-51a4-416c-80ee-de4408f10f4d" path="/var/lib/kubelet/pods/1fcc284b-51a4-416c-80ee-de4408f10f4d/volumes" Nov 02 20:40:22.274208 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:22.270597 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:25.196710 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:25.196658 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:25.990306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:25.990281 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:40:26.000233 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.000200 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-data\") pod \"fee3004b-ec0b-4c68-bcab-23b727eb7838\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " Nov 02 20:40:26.000395 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.000286 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-workload-socket\") pod \"fee3004b-ec0b-4c68-bcab-23b727eb7838\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " Nov 02 20:40:26.000443 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.000416 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-envoy\") pod \"fee3004b-ec0b-4c68-bcab-23b727eb7838\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " Nov 02 20:40:26.000479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.000463 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-data" pod "fee3004b-ec0b-4c68-bcab-23b727eb7838" (UID: "fee3004b-ec0b-4c68-bcab-23b727eb7838"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:26.000515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.000477 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee3004b-ec0b-4c68-bcab-23b727eb7838-kube-api-access-td6r7\" (UniqueName: \"kubernetes.io/projected/fee3004b-ec0b-4c68-bcab-23b727eb7838-kube-api-access-td6r7\") pod \"fee3004b-ec0b-4c68-bcab-23b727eb7838\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " Nov 02 20:40:26.000549 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.000529 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-workload-socket" pod "fee3004b-ec0b-4c68-bcab-23b727eb7838" (UID: "fee3004b-ec0b-4c68-bcab-23b727eb7838"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:26.000586 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.000566 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-podinfo\") pod \"fee3004b-ec0b-4c68-bcab-23b727eb7838\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " Nov 02 20:40:26.000645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.000630 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-workload-certs\") pod \"fee3004b-ec0b-4c68-bcab-23b727eb7838\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " Nov 02 20:40:26.000678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.000669 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fee3004b-ec0b-4c68-bcab-23b727eb7838-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fee3004b-ec0b-4c68-bcab-23b727eb7838-istiod-ca-cert\") pod \"fee3004b-ec0b-4c68-bcab-23b727eb7838\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " Nov 02 20:40:26.000715 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.000694 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-token\" (UniqueName: \"kubernetes.io/projected/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-token\") pod \"fee3004b-ec0b-4c68-bcab-23b727eb7838\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " Nov 02 20:40:26.000828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.000777 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-credential-socket\") pod \"fee3004b-ec0b-4c68-bcab-23b727eb7838\" (UID: \"fee3004b-ec0b-4c68-bcab-23b727eb7838\") " Nov 02 20:40:26.001294 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.000945 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-workload-certs" pod "fee3004b-ec0b-4c68-bcab-23b727eb7838" (UID: "fee3004b-ec0b-4c68-bcab-23b727eb7838"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:26.001294 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.001124 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-credential-socket" pod "fee3004b-ec0b-4c68-bcab-23b727eb7838" (UID: "fee3004b-ec0b-4c68-bcab-23b727eb7838"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:26.001294 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.001168 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee3004b-ec0b-4c68-bcab-23b727eb7838-istiod-ca-cert" pod "fee3004b-ec0b-4c68-bcab-23b727eb7838" (UID: "fee3004b-ec0b-4c68-bcab-23b727eb7838"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:40:26.001892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.001864 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:26.001993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.001898 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:26.001993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.001912 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:26.001993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.001926 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fee3004b-ec0b-4c68-bcab-23b727eb7838-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:26.001993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.001942 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:26.003108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.003065 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-envoy" pod "fee3004b-ec0b-4c68-bcab-23b727eb7838" (UID: "fee3004b-ec0b-4c68-bcab-23b727eb7838"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:26.003409 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.003390 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-podinfo" pod "fee3004b-ec0b-4c68-bcab-23b727eb7838" (UID: "fee3004b-ec0b-4c68-bcab-23b727eb7838"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:40:26.003476 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.003456 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee3004b-ec0b-4c68-bcab-23b727eb7838-kube-api-access-td6r7" pod "fee3004b-ec0b-4c68-bcab-23b727eb7838" (UID: "fee3004b-ec0b-4c68-bcab-23b727eb7838"). InnerVolumeSpecName "kube-api-access-td6r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:40:26.003668 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.003652 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-token" pod "fee3004b-ec0b-4c68-bcab-23b727eb7838" (UID: "fee3004b-ec0b-4c68-bcab-23b727eb7838"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:40:26.048222 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.048175 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:26.103429 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.103402 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:26.103429 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.103427 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td6r7\" (UniqueName: \"kubernetes.io/projected/fee3004b-ec0b-4c68-bcab-23b727eb7838-kube-api-access-td6r7\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:26.103588 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.103439 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:26.103588 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.103449 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fee3004b-ec0b-4c68-bcab-23b727eb7838-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:26.312706 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.312611 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:26.686342 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.686310 4971 generic.go:356] "Generic (PLEG): container finished" podID="fee3004b-ec0b-4c68-bcab-23b727eb7838" containerID="a301638ef3cc6d42ebdd0ad9f6c7e555a221d6b755a23c2dbc84b692a6f3680e" exitCode=0 Nov 02 20:40:26.686507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.686371 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" Nov 02 20:40:26.686507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.686365 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" event={"ID":"fee3004b-ec0b-4c68-bcab-23b727eb7838","Type":"ContainerDied","Data":"a301638ef3cc6d42ebdd0ad9f6c7e555a221d6b755a23c2dbc84b692a6f3680e"} Nov 02 20:40:26.686507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.686444 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7" event={"ID":"fee3004b-ec0b-4c68-bcab-23b727eb7838","Type":"ContainerDied","Data":"9b80c6ac7114933811187734df5098d2e900ba35beac9ff74750f90c8d02461a"} Nov 02 20:40:26.686507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.686476 4971 scope.go:122] "RemoveContainer" containerID="a301638ef3cc6d42ebdd0ad9f6c7e555a221d6b755a23c2dbc84b692a6f3680e" Nov 02 20:40:26.701502 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.701482 4971 scope.go:122] "RemoveContainer" containerID="8ba9cee98bed6ceb1b84bc49b0dd24f01f1ea01fb5cb42b0135ab93aebc54641" Nov 02 20:40:26.714163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.714141 4971 scope.go:122] "RemoveContainer" containerID="a301638ef3cc6d42ebdd0ad9f6c7e555a221d6b755a23c2dbc84b692a6f3680e" Nov 02 20:40:26.714511 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:40:26.714485 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a301638ef3cc6d42ebdd0ad9f6c7e555a221d6b755a23c2dbc84b692a6f3680e\": container with ID starting with a301638ef3cc6d42ebdd0ad9f6c7e555a221d6b755a23c2dbc84b692a6f3680e not found: ID does not exist" containerID="a301638ef3cc6d42ebdd0ad9f6c7e555a221d6b755a23c2dbc84b692a6f3680e" Nov 02 20:40:26.714598 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.714523 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a301638ef3cc6d42ebdd0ad9f6c7e555a221d6b755a23c2dbc84b692a6f3680e"} err="failed to get container status \"a301638ef3cc6d42ebdd0ad9f6c7e555a221d6b755a23c2dbc84b692a6f3680e\": rpc error: code = NotFound desc = could not find container \"a301638ef3cc6d42ebdd0ad9f6c7e555a221d6b755a23c2dbc84b692a6f3680e\": container with ID starting with a301638ef3cc6d42ebdd0ad9f6c7e555a221d6b755a23c2dbc84b692a6f3680e not found: ID does not exist" Nov 02 20:40:26.714598 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.714548 4971 scope.go:122] "RemoveContainer" containerID="8ba9cee98bed6ceb1b84bc49b0dd24f01f1ea01fb5cb42b0135ab93aebc54641" Nov 02 20:40:26.714881 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:40:26.714861 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba9cee98bed6ceb1b84bc49b0dd24f01f1ea01fb5cb42b0135ab93aebc54641\": container with ID starting with 8ba9cee98bed6ceb1b84bc49b0dd24f01f1ea01fb5cb42b0135ab93aebc54641 not found: ID does not exist" containerID="8ba9cee98bed6ceb1b84bc49b0dd24f01f1ea01fb5cb42b0135ab93aebc54641" Nov 02 20:40:26.714953 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.714887 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba9cee98bed6ceb1b84bc49b0dd24f01f1ea01fb5cb42b0135ab93aebc54641"} err="failed to get container status \"8ba9cee98bed6ceb1b84bc49b0dd24f01f1ea01fb5cb42b0135ab93aebc54641\": rpc error: code = NotFound desc = could not find container \"8ba9cee98bed6ceb1b84bc49b0dd24f01f1ea01fb5cb42b0135ab93aebc54641\": container with ID starting with 8ba9cee98bed6ceb1b84bc49b0dd24f01f1ea01fb5cb42b0135ab93aebc54641 not found: ID does not exist" Nov 02 20:40:26.846169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.846143 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7"] Nov 02 20:40:26.892472 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:26.892447 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-6c66f658-gw-openshift-default-d46475fc-kx2j7"] Nov 02 20:40:27.182558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:27.182514 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:27.437333 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:27.436715 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:27.613106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:27.610498 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r"] Nov 02 20:40:27.624681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:27.622420 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-574357e2-s964r"] Nov 02 20:40:27.682447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:27.682407 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:27.685842 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:27.685818 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65"] Nov 02 20:40:27.686053 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:27.686029 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" podUID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" containerName="istio-proxy" containerID="cri-o://c881f3cc4e3421e08aa6a1a4cc9924e4a948a9e43f140b3c40a3515a27ca9323" gracePeriod=30 Nov 02 20:40:27.886105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:27.885074 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a39c6b-dff3-4909-b2ef-1126c3ab8281" path="/var/lib/kubelet/pods/44a39c6b-dff3-4909-b2ef-1126c3ab8281/volumes" Nov 02 20:40:27.886105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:27.885916 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee3004b-ec0b-4c68-bcab-23b727eb7838" path="/var/lib/kubelet/pods/fee3004b-ec0b-4c68-bcab-23b727eb7838/volumes" Nov 02 20:40:27.931112 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:27.930547 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:28.792638 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:28.792604 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:29.110422 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:29.110390 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:29.610987 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:29.610961 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" Nov 02 20:40:29.775332 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:29.775288 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:30.022694 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:30.022599 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:30.639292 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:30.639261 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:31.051564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:31.051490 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s"] Nov 02 20:40:31.097702 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:31.097672 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b9a426-mw79s"] Nov 02 20:40:31.870703 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:31.870668 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="656598b3-b015-49b7-b955-55cd54861736" path="/var/lib/kubelet/pods/656598b3-b015-49b7-b955-55cd54861736/volumes" Nov 02 20:40:32.242285 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:32.242183 4971 patch_prober.go:29] interesting pod/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:40:32.242285 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:32.242271 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" podUID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:40:32.681820 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:32.681759 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:32.997018 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:32.996985 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:40:33.045665 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.045639 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-credential-socket\") pod \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " Nov 02 20:40:33.045807 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.045732 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-workload-certs\") pod \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " Nov 02 20:40:33.045807 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.045805 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-workload-socket\") pod \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " Nov 02 20:40:33.045891 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.045833 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-data\") pod \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " Nov 02 20:40:33.045941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.045892 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-token\" (UniqueName: \"kubernetes.io/projected/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-token\") pod \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " Nov 02 20:40:33.045993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.045939 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-credential-socket" pod "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" (UID: "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:33.045993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.045949 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-workload-certs" pod "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" (UID: "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:33.045993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.045949 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istiod-ca-cert\") pod \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " Nov 02 20:40:33.046158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.046039 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-podinfo\") pod \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " Nov 02 20:40:33.046158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.046076 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-envoy\") pod \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " Nov 02 20:40:33.046508 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.046241 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-data" pod "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" (UID: "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:33.046508 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.046273 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-workload-socket" pod "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" (UID: "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:33.046508 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.046340 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-kube-api-access-f8wbf\" (UniqueName: \"kubernetes.io/projected/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-kube-api-access-f8wbf\") pod \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\" (UID: \"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a\") " Nov 02 20:40:33.046508 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.046431 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istiod-ca-cert" pod "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" (UID: "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:40:33.047347 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.047321 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:33.047455 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.047379 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:33.047455 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.047396 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:33.047455 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.047411 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:33.047455 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.047425 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:33.048498 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.048467 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-token" pod "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" (UID: "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:40:33.048612 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.048524 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-podinfo" pod "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" (UID: "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:40:33.048612 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.048589 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-kube-api-access-f8wbf" pod "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" (UID: "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a"). InnerVolumeSpecName "kube-api-access-f8wbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:40:33.048731 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.048621 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-envoy" pod "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" (UID: "c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:40:33.148610 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.148579 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8wbf\" (UniqueName: \"kubernetes.io/projected/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-kube-api-access-f8wbf\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:33.148610 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.148610 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:33.148761 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.148621 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:33.148761 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.148630 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:33.340245 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:40:33.340164 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:40:33.755373 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.755340 4971 generic.go:356] "Generic (PLEG): container finished" podID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" containerID="c881f3cc4e3421e08aa6a1a4cc9924e4a948a9e43f140b3c40a3515a27ca9323" exitCode=0 Nov 02 20:40:33.755609 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.755405 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" Nov 02 20:40:33.755609 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.755453 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" event={"ID":"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a","Type":"ContainerDied","Data":"c881f3cc4e3421e08aa6a1a4cc9924e4a948a9e43f140b3c40a3515a27ca9323"} Nov 02 20:40:33.755609 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.755546 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65" event={"ID":"c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a","Type":"ContainerDied","Data":"96dcb7fc94e658c65f5fe915222b90d7b7b206288d9da6833a231c4e45c100cb"} Nov 02 20:40:33.755609 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.755576 4971 scope.go:122] "RemoveContainer" containerID="c881f3cc4e3421e08aa6a1a4cc9924e4a948a9e43f140b3c40a3515a27ca9323" Nov 02 20:40:33.768411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.768396 4971 scope.go:122] "RemoveContainer" containerID="123686016c6f8ee17ad443eb47bce41508ba670fb3e6b680aa1926e8abf5fca7" Nov 02 20:40:33.781287 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.781248 4971 scope.go:122] "RemoveContainer" containerID="c881f3cc4e3421e08aa6a1a4cc9924e4a948a9e43f140b3c40a3515a27ca9323" Nov 02 20:40:33.781561 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:40:33.781541 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c881f3cc4e3421e08aa6a1a4cc9924e4a948a9e43f140b3c40a3515a27ca9323\": container with ID starting with c881f3cc4e3421e08aa6a1a4cc9924e4a948a9e43f140b3c40a3515a27ca9323 not found: ID does not exist" containerID="c881f3cc4e3421e08aa6a1a4cc9924e4a948a9e43f140b3c40a3515a27ca9323" Nov 02 20:40:33.781629 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.781570 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c881f3cc4e3421e08aa6a1a4cc9924e4a948a9e43f140b3c40a3515a27ca9323"} err="failed to get container status \"c881f3cc4e3421e08aa6a1a4cc9924e4a948a9e43f140b3c40a3515a27ca9323\": rpc error: code = NotFound desc = could not find container \"c881f3cc4e3421e08aa6a1a4cc9924e4a948a9e43f140b3c40a3515a27ca9323\": container with ID starting with c881f3cc4e3421e08aa6a1a4cc9924e4a948a9e43f140b3c40a3515a27ca9323 not found: ID does not exist" Nov 02 20:40:33.781629 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.781587 4971 scope.go:122] "RemoveContainer" containerID="123686016c6f8ee17ad443eb47bce41508ba670fb3e6b680aa1926e8abf5fca7" Nov 02 20:40:33.781881 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:40:33.781852 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"123686016c6f8ee17ad443eb47bce41508ba670fb3e6b680aa1926e8abf5fca7\": container with ID starting with 123686016c6f8ee17ad443eb47bce41508ba670fb3e6b680aa1926e8abf5fca7 not found: ID does not exist" containerID="123686016c6f8ee17ad443eb47bce41508ba670fb3e6b680aa1926e8abf5fca7" Nov 02 20:40:33.781943 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.781890 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"123686016c6f8ee17ad443eb47bce41508ba670fb3e6b680aa1926e8abf5fca7"} err="failed to get container status \"123686016c6f8ee17ad443eb47bce41508ba670fb3e6b680aa1926e8abf5fca7\": rpc error: code = NotFound desc = could not find container \"123686016c6f8ee17ad443eb47bce41508ba670fb3e6b680aa1926e8abf5fca7\": container with ID starting with 123686016c6f8ee17ad443eb47bce41508ba670fb3e6b680aa1926e8abf5fca7 not found: ID does not exist" Nov 02 20:40:33.914993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.914965 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65"] Nov 02 20:40:33.961576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:33.961553 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-574357e2-openshift-default-6b6d6c796f-4nf65"] Nov 02 20:40:35.868666 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:35.868633 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" path="/var/lib/kubelet/pods/c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a/volumes" Nov 02 20:40:36.781358 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.781329 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m"] Nov 02 20:40:36.782070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.782053 4971 cpu_manager.go:414] "containerMap: removing container" podUID="fee3004b-ec0b-4c68-bcab-23b727eb7838" containerName="istio-proxy" Nov 02 20:40:36.782156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.782071 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee3004b-ec0b-4c68-bcab-23b727eb7838" containerName="istio-proxy" Nov 02 20:40:36.782156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.782099 4971 cpu_manager.go:414] "containerMap: removing container" podUID="fee3004b-ec0b-4c68-bcab-23b727eb7838" containerName="istio-proxy" Nov 02 20:40:36.782156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.782105 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee3004b-ec0b-4c68-bcab-23b727eb7838" containerName="istio-proxy" Nov 02 20:40:36.782156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.782131 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" containerName="istio-proxy" Nov 02 20:40:36.782156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.782137 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" containerName="istio-proxy" Nov 02 20:40:36.782156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.782147 4971 cpu_manager.go:414] "containerMap: removing container" podUID="5847fbc4-1639-4ed1-ae14-182e5ab8b50f" containerName="istio-proxy" Nov 02 20:40:36.782156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.782151 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5847fbc4-1639-4ed1-ae14-182e5ab8b50f" containerName="istio-proxy" Nov 02 20:40:36.782379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.782296 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee3004b-ec0b-4c68-bcab-23b727eb7838" containerName="istio-proxy" Nov 02 20:40:36.782379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.782304 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee3004b-ec0b-4c68-bcab-23b727eb7838" containerName="istio-proxy" Nov 02 20:40:36.782379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.782316 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" containerName="istio-proxy" Nov 02 20:40:36.782379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.782323 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" containerName="istio-proxy" Nov 02 20:40:36.786632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.786612 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:36.836300 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.836274 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 20:40:36.836417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.836325 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qhb9c\"" Nov 02 20:40:36.836417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.836325 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 20:40:36.836417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.836353 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 20:40:36.889370 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.889346 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m"] Nov 02 20:40:36.900796 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.900770 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:36.900918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.900812 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:36.900918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.900847 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbspp\" (UniqueName: \"kubernetes.io/projected/181451b7-ca86-4aa2-ae78-73f17b5a67d2-kube-api-access-rbspp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:36.901062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.901038 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:36.901158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.901139 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:36.901265 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.901237 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:36.901330 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:36.901304 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/181451b7-ca86-4aa2-ae78-73f17b5a67d2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:37.002634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.002602 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:37.002804 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.002657 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/181451b7-ca86-4aa2-ae78-73f17b5a67d2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:37.003103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.002908 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:37.003103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.002952 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:37.003103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.002983 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbspp\" (UniqueName: \"kubernetes.io/projected/181451b7-ca86-4aa2-ae78-73f17b5a67d2-kube-api-access-rbspp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:37.003293 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.003108 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:37.003293 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.003149 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:37.003442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.003424 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:37.006483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.006457 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:37.006610 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.006523 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/181451b7-ca86-4aa2-ae78-73f17b5a67d2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:37.006703 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.006680 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:37.009031 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.009002 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:37.061006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.060949 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbspp\" (UniqueName: \"kubernetes.io/projected/181451b7-ca86-4aa2-ae78-73f17b5a67d2-kube-api-access-rbspp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:37.061169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.061150 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sl97m\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:37.099405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.099384 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:40:37.466247 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.466215 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:37.528247 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.528214 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m"] Nov 02 20:40:37.556157 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:40:37.556068 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181451b7_ca86_4aa2_ae78_73f17b5a67d2.slice/crio-709e52eff676a060490332bb9dc7f5940d0b0b729bc502dfb1d5895adb57fc9f WatchSource:0}: Error finding container 709e52eff676a060490332bb9dc7f5940d0b0b729bc502dfb1d5895adb57fc9f: Status 404 returned error can't find the container with id 709e52eff676a060490332bb9dc7f5940d0b0b729bc502dfb1d5895adb57fc9f Nov 02 20:40:37.795816 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:37.795738 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" event={"ID":"181451b7-ca86-4aa2-ae78-73f17b5a67d2","Type":"ContainerStarted","Data":"709e52eff676a060490332bb9dc7f5940d0b0b729bc502dfb1d5895adb57fc9f"} Nov 02 20:40:39.770971 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:39.770936 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:42.305624 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:42.305588 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:46.050742 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:46.050711 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n"] Nov 02 20:40:46.097092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:46.097053 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3ea48cd-j8h4n"] Nov 02 20:40:47.107669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:47.107625 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:47.868794 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:47.868761 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff22da18-e34e-4735-ac05-daae7268fb06" path="/var/lib/kubelet/pods/ff22da18-e34e-4735-ac05-daae7268fb06/volumes" Nov 02 20:40:51.899324 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:51.896849 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:56.049295 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:56.049260 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr"] Nov 02 20:40:56.095942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:56.095911 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de529379-dlznr"] Nov 02 20:40:56.593727 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:56.593691 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:56.844836 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:56.844751 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:40:57.869810 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:40:57.869775 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8af1ff-e779-4317-b2f3-e9372d6e3a31" path="/var/lib/kubelet/pods/fd8af1ff-e779-4317-b2f3-e9372d6e3a31/volumes" Nov 02 20:41:01.150186 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:01.150148 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:41:01.150469 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:01.150224 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:41:01.599641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:01.599607 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:02.026965 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:02.026852 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" event={"ID":"181451b7-ca86-4aa2-ae78-73f17b5a67d2","Type":"ContainerStarted","Data":"8a77e8df154012a822ad3e97f8e1dabda47497f25cc95dadd4a3c4df43bee371"} Nov 02 20:41:02.027226 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:02.027176 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:41:02.029886 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:02.029857 4971 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-sl97m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:41:02.030044 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:02.029914 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" podUID="181451b7-ca86-4aa2-ae78-73f17b5a67d2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:41:02.230779 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:02.230724 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" podStartSLOduration=2.640447309 podStartE2EDuration="26.23070991s" totalImagesPullingTime="23.590262601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:41:02.189392059 +0000 UTC m=+1590.479048588" watchObservedRunningTime="2026-11-02 20:41:02.23070991 +0000 UTC m=+1590.520366432" Nov 02 20:41:03.036116 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:03.036068 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:41:03.351260 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:41:03.351225 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:41:06.561944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:06.561912 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:09.337019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:09.336986 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:10.056858 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:10.056821 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx"] Nov 02 20:41:10.099795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:10.099751 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c558b71-6fgxx"] Nov 02 20:41:10.675524 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:10.675483 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:10.944503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:10.944380 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:11.491512 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:11.491471 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:11.750453 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:11.750360 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:11.870126 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:11.870072 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf57a6d-167e-46ce-a278-dfdfa345c3e8" path="/var/lib/kubelet/pods/3bf57a6d-167e-46ce-a278-dfdfa345c3e8/volumes" Nov 02 20:41:12.048626 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:12.048553 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w"] Nov 02 20:41:12.094859 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:12.094832 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbf98b2e-zcl9w"] Nov 02 20:41:13.868599 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:13.868568 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a48e90-9a8b-4a12-95e9-0b5fc5232699" path="/var/lib/kubelet/pods/f9a48e90-9a8b-4a12-95e9-0b5fc5232699/volumes" Nov 02 20:41:15.049328 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:15.049295 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw"] Nov 02 20:41:15.095938 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:15.095897 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cba244ab-dwnsw"] Nov 02 20:41:15.146999 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:15.146963 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:15.392927 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:15.392894 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:15.884239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:15.884207 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb55d37-552a-44cc-bc14-0e4c1addfb3f" path="/var/lib/kubelet/pods/6bb55d37-552a-44cc-bc14-0e4c1addfb3f/volumes" Nov 02 20:41:17.063505 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:17.063468 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:20.220833 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:20.220795 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:20.498954 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:20.498866 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:21.219504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.219475 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q"] Nov 02 20:41:21.220231 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.220216 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" containerName="istio-proxy" Nov 02 20:41:21.220280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.220233 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c221b7fd-3fe6-4ddc-b09e-38eb310e9d7a" containerName="istio-proxy" Nov 02 20:41:21.223504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.223487 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q" Nov 02 20:41:21.278278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.278249 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q"] Nov 02 20:41:21.316809 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.316781 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:41:21.317414 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.317394 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:41:21.419965 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.419928 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76782b5d-wxk4q\" (UID: \"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q" Nov 02 20:41:21.420191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.419973 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf8sb\" (UniqueName: \"kubernetes.io/projected/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-kube-api-access-kf8sb\") pod \"maas-api-revoke-sub-76782b5d-wxk4q\" (UID: \"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q" Nov 02 20:41:21.420191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.420160 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76782b5d-wxk4q\" (UID: \"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q" Nov 02 20:41:21.521473 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.521389 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76782b5d-wxk4q\" (UID: \"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q" Nov 02 20:41:21.521473 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.521431 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf8sb\" (UniqueName: \"kubernetes.io/projected/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-kube-api-access-kf8sb\") pod \"maas-api-revoke-sub-76782b5d-wxk4q\" (UID: \"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q" Nov 02 20:41:21.521473 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.521475 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76782b5d-wxk4q\" (UID: \"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q" Nov 02 20:41:21.522230 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.522209 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76782b5d-wxk4q\" (UID: \"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q" Nov 02 20:41:21.523877 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.523854 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76782b5d-wxk4q\" (UID: \"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q" Nov 02 20:41:21.579434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.579400 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf8sb\" (UniqueName: \"kubernetes.io/projected/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-kube-api-access-kf8sb\") pod \"maas-api-revoke-sub-76782b5d-wxk4q\" (UID: \"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q" Nov 02 20:41:21.838461 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:21.838369 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q" Nov 02 20:41:22.080785 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:22.080744 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:22.221029 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:41:22.220997 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e7e19d_06dd_46c1_b35d_645c9f1f02f3.slice/crio-0ca7155ac7217d0baa25dd4c10230b84a166ae0e8e20b9da6e22bad7e6117f26 WatchSource:0}: Error finding container 0ca7155ac7217d0baa25dd4c10230b84a166ae0e8e20b9da6e22bad7e6117f26: Status 404 returned error can't find the container with id 0ca7155ac7217d0baa25dd4c10230b84a166ae0e8e20b9da6e22bad7e6117f26 Nov 02 20:41:22.223107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:22.223066 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q"] Nov 02 20:41:22.507642 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:22.507545 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q" event={"ID":"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3","Type":"ContainerStarted","Data":"0ca7155ac7217d0baa25dd4c10230b84a166ae0e8e20b9da6e22bad7e6117f26"} Nov 02 20:41:23.520539 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:23.520451 4971 generic.go:356] "Generic (PLEG): container finished" podID="b5e7e19d-06dd-46c1-b35d-645c9f1f02f3" containerID="fa6707ae85f40c9eaae5dba551b7f8a0e15b6dbc348e970d562e0f1ad7753f5e" exitCode=0 Nov 02 20:41:23.520906 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:23.520536 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q" event={"ID":"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3","Type":"ContainerDied","Data":"fa6707ae85f40c9eaae5dba551b7f8a0e15b6dbc348e970d562e0f1ad7753f5e"} Nov 02 20:41:24.178488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:24.178460 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:24.900622 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:24.900592 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q" Nov 02 20:41:25.038852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.038825 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-kube-api-access-kf8sb\" (UniqueName: \"kubernetes.io/projected/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-kube-api-access-kf8sb\") pod \"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3\" (UID: \"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3\") " Nov 02 20:41:25.045269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.038917 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-maas-api-xks-ca\") pod \"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3\" (UID: \"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3\") " Nov 02 20:41:25.045269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.039106 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-maas-api-openshift-ca\") pod \"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3\" (UID: \"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3\") " Nov 02 20:41:25.045269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.039468 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-maas-api-openshift-ca" pod "b5e7e19d-06dd-46c1-b35d-645c9f1f02f3" (UID: "b5e7e19d-06dd-46c1-b35d-645c9f1f02f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:41:25.045269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.040723 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:25.045269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.041630 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-maas-api-xks-ca" pod "b5e7e19d-06dd-46c1-b35d-645c9f1f02f3" (UID: "b5e7e19d-06dd-46c1-b35d-645c9f1f02f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:41:25.045269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.041914 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-kube-api-access-kf8sb" pod "b5e7e19d-06dd-46c1-b35d-645c9f1f02f3" (UID: "b5e7e19d-06dd-46c1-b35d-645c9f1f02f3"). InnerVolumeSpecName "kube-api-access-kf8sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:41:25.142768 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.142742 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:25.142768 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.142769 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf8sb\" (UniqueName: \"kubernetes.io/projected/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3-kube-api-access-kf8sb\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:25.544492 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.544413 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q" Nov 02 20:41:25.544492 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.544439 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q" event={"ID":"b5e7e19d-06dd-46c1-b35d-645c9f1f02f3","Type":"ContainerDied","Data":"0ca7155ac7217d0baa25dd4c10230b84a166ae0e8e20b9da6e22bad7e6117f26"} Nov 02 20:41:25.544492 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.544489 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ca7155ac7217d0baa25dd4c10230b84a166ae0e8e20b9da6e22bad7e6117f26" Nov 02 20:41:25.546421 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.546392 4971 generic.go:356] "Generic (PLEG): container finished" podID="a50c847f-3e10-42a9-8ca1-1e285db2af51" containerID="cd843bc1894ec2b4d1b846cf3045d9c6371356d9a2e5abccf93ffd69a7e6ec46" exitCode=0 Nov 02 20:41:25.546573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.546470 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" event={"ID":"a50c847f-3e10-42a9-8ca1-1e285db2af51","Type":"ContainerDied","Data":"cd843bc1894ec2b4d1b846cf3045d9c6371356d9a2e5abccf93ffd69a7e6ec46"} Nov 02 20:41:25.546863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.546847 4971 scope.go:122] "RemoveContainer" containerID="cd843bc1894ec2b4d1b846cf3045d9c6371356d9a2e5abccf93ffd69a7e6ec46" Nov 02 20:41:25.548535 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.548503 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:41:25.548660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.548572 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:41:25.548660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.548604 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:41:25.818154 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:25.817541 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:26.062998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:26.062963 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:26.558408 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:26.558370 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" event={"ID":"a50c847f-3e10-42a9-8ca1-1e285db2af51","Type":"ContainerStarted","Data":"ca9a2dc23966395329b8e7ab0951697da61a862074071bb33b161e21d028ddeb"} Nov 02 20:41:26.659409 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:26.659372 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:41:26.661006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:26.660974 4971 patch_prober.go:29] interesting pod/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 20:41:26.661158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:26.661030 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" podUID="a50c847f-3e10-42a9-8ca1-1e285db2af51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 20:41:27.103287 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:27.103249 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:27.658993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:27.658958 4971 patch_prober.go:29] interesting pod/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 20:41:27.659209 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:27.659010 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" podUID="a50c847f-3e10-42a9-8ca1-1e285db2af51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 20:41:28.509741 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:28.509704 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:41:28.659553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:28.659524 4971 patch_prober.go:29] interesting pod/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 20:41:28.659721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:28.659588 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" podUID="a50c847f-3e10-42a9-8ca1-1e285db2af51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 20:41:29.659810 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:29.659777 4971 patch_prober.go:29] interesting pod/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 20:41:29.660234 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:29.659827 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" podUID="a50c847f-3e10-42a9-8ca1-1e285db2af51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 20:41:29.864167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:29.864135 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:30.160545 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:30.160514 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:30.659640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:30.659607 4971 patch_prober.go:29] interesting pod/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 20:41:30.659809 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:30.659661 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" podUID="a50c847f-3e10-42a9-8ca1-1e285db2af51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 20:41:31.001720 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:31.001646 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:31.053752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:31.053723 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq"] Nov 02 20:41:31.096966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:31.096930 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7272cc2b-g8bwq"] Nov 02 20:41:31.659436 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:31.659398 4971 patch_prober.go:29] interesting pod/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 20:41:31.659643 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:31.659463 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" podUID="a50c847f-3e10-42a9-8ca1-1e285db2af51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 20:41:31.869540 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:31.869505 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3781dc44-be73-4b0c-8a66-dcb8c8ffc07c" path="/var/lib/kubelet/pods/3781dc44-be73-4b0c-8a66-dcb8c8ffc07c/volumes" Nov 02 20:41:32.288358 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:32.288322 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:32.531122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:32.531051 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:32.663872 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:32.663846 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:41:32.665144 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:32.665074 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:41:33.049149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:33.049045 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf"] Nov 02 20:41:33.054606 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:33.054578 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8034c8ec-gwhmf"] Nov 02 20:41:33.132746 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:33.132699 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:33.369074 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:41:33.369025 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:41:33.388613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:33.388534 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:33.900108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:33.893238 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c09a3528-b25d-40a2-8df7-14e997368a0a" path="/var/lib/kubelet/pods/c09a3528-b25d-40a2-8df7-14e997368a0a/volumes" Nov 02 20:41:35.052807 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:35.052777 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f"] Nov 02 20:41:35.099449 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:35.099416 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406ef455-dsl7f"] Nov 02 20:41:35.870002 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:35.869970 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9" path="/var/lib/kubelet/pods/2eaf0d3d-a203-4b8d-ab64-5d6a5f114ce9/volumes" Nov 02 20:41:36.059881 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:36.059849 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw"] Nov 02 20:41:36.064999 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:36.064976 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a1ba2f-nk8cw"] Nov 02 20:41:37.371137 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:37.371075 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:37.868824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:37.868796 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a4cb21-606f-4b21-819d-9e312d0fce5f" path="/var/lib/kubelet/pods/60a4cb21-606f-4b21-819d-9e312d0fce5f/volumes" Nov 02 20:41:38.311748 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:38.311642 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:39.047600 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:39.047572 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p"] Nov 02 20:41:39.093878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:39.093849 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef946bb-ccq6p"] Nov 02 20:41:39.868530 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:39.868485 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a94a51-90f5-4f9b-9118-3c1a85eb8690" path="/var/lib/kubelet/pods/67a94a51-90f5-4f9b-9118-3c1a85eb8690/volumes" Nov 02 20:41:41.150622 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:41.150591 4971 scope.go:122] "RemoveContainer" containerID="91c4a62be57106d893f737d8aa75e98725237cf0bd1f6d0b7d3bce3a134dfbe6" Nov 02 20:41:41.162404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:41.162381 4971 scope.go:122] "RemoveContainer" containerID="26ca336cd2d192736ffb46188c94d7f294f064e3a34d3888b14f809a1bef5e50" Nov 02 20:41:42.233189 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:42.233155 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:47.238240 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:47.238190 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:41:47.316108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:47.315445 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 20:41:47.321117 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:47.316640 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:41:47.331108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:47.327349 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt" podUID="5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 02 20:41:47.433273 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:47.433203 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded" Nov 02 20:41:47.455509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:47.455474 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:41:47.468604 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:47.468574 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" probeResult="failure" output="Get \"https://10.132.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:41:47.488183 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:47.488138 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:41:47.576018 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:47.575920 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:47.732233 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:47.732169 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:41:47.804564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:47.804523 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:41:48.018789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:48.018735 4971 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:41:49.796270 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:49.796234 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:50.050656 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:50.050575 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn"] Nov 02 20:41:50.053492 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:50.053462 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:50.097698 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:50.097671 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e45148-2d2hn"] Nov 02 20:41:50.103651 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:50.103622 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv"] Nov 02 20:41:50.108273 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:50.108252 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edac9fbd-swqnv"] Nov 02 20:41:51.915674 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:51.915632 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09619f70-5405-41ed-94fd-5f6f5e0fe0f6" path="/var/lib/kubelet/pods/09619f70-5405-41ed-94fd-5f6f5e0fe0f6/volumes" Nov 02 20:41:51.916395 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:51.916371 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53925b8-a1db-4110-a978-8a32f6f445ed" path="/var/lib/kubelet/pods/a53925b8-a1db-4110-a978-8a32f6f445ed/volumes" Nov 02 20:41:52.806543 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:52.806507 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:53.505642 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:53.505607 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75"] Nov 02 20:41:53.506120 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:53.505834 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" podUID="f0be871c-832b-4e45-965c-d092b43030eb" containerName="istio-proxy" containerID="cri-o://f56e4dddc8c2f2f840507acd69acac1751371c4a8e581c1eed6d2dade80389fd" gracePeriod=30 Nov 02 20:41:53.927274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:53.927228 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:54.049654 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:54.049612 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw"] Nov 02 20:41:54.054901 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:54.054871 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea955427-5fhsw"] Nov 02 20:41:55.049033 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:55.048999 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc"] Nov 02 20:41:55.095663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:55.095630 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba4233da-lh6qc"] Nov 02 20:41:55.868859 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:55.868826 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95853df-8304-4539-8e10-60a57f2ab497" path="/var/lib/kubelet/pods/d95853df-8304-4539-8e10-60a57f2ab497/volumes" Nov 02 20:41:55.869435 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:55.869413 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0981e22-b642-418b-9a57-482a92589ec4" path="/var/lib/kubelet/pods/f0981e22-b642-418b-9a57-482a92589ec4/volumes" Nov 02 20:41:57.006025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.005998 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh"] Nov 02 20:41:57.006750 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.006733 4971 cpu_manager.go:414] "containerMap: removing container" podUID="b5e7e19d-06dd-46c1-b35d-645c9f1f02f3" containerName="revoke-keys" Nov 02 20:41:57.006795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.006751 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e7e19d-06dd-46c1-b35d-645c9f1f02f3" containerName="revoke-keys" Nov 02 20:41:57.006919 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.006909 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e7e19d-06dd-46c1-b35d-645c9f1f02f3" containerName="revoke-keys" Nov 02 20:41:57.012165 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.012145 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" Nov 02 20:41:57.077956 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.077916 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh"] Nov 02 20:41:57.087076 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.087041 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-df17143f-nckbh\" (UID: \"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" Nov 02 20:41:57.087322 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.087300 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-df17143f-nckbh\" (UID: \"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" Nov 02 20:41:57.087597 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.087581 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42wx\" (UniqueName: \"kubernetes.io/projected/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-kube-api-access-x42wx\") pod \"maas-api-revoke-keys-e2e-ait-df17143f-nckbh\" (UID: \"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" Nov 02 20:41:57.104533 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.104508 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:41:57.104725 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.104639 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:41:57.189114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.189060 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42wx\" (UniqueName: \"kubernetes.io/projected/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-kube-api-access-x42wx\") pod \"maas-api-revoke-keys-e2e-ait-df17143f-nckbh\" (UID: \"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" Nov 02 20:41:57.189269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.189254 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-df17143f-nckbh\" (UID: \"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" Nov 02 20:41:57.189314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.189287 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-df17143f-nckbh\" (UID: \"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" Nov 02 20:41:57.189958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.189910 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-df17143f-nckbh\" (UID: \"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" Nov 02 20:41:57.191810 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.191788 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-df17143f-nckbh\" (UID: \"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" Nov 02 20:41:57.247318 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.247290 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42wx\" (UniqueName: \"kubernetes.io/projected/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-kube-api-access-x42wx\") pod \"maas-api-revoke-keys-e2e-ait-df17143f-nckbh\" (UID: \"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" Nov 02 20:41:57.329471 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.329408 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" Nov 02 20:41:57.710760 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:41:57.710727 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode084e043_2e14_4d59_b82e_6f6b1c8aa3b1.slice/crio-41488a702b963da49d3fabbf58667458c01643dabb1a2371239b4e476156b2e6 WatchSource:0}: Error finding container 41488a702b963da49d3fabbf58667458c01643dabb1a2371239b4e476156b2e6: Status 404 returned error can't find the container with id 41488a702b963da49d3fabbf58667458c01643dabb1a2371239b4e476156b2e6 Nov 02 20:41:57.712338 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:57.712311 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh"] Nov 02 20:41:58.145970 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.145913 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:41:58.228222 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.228186 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" event={"ID":"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1","Type":"ContainerStarted","Data":"e81064b4a36e66691fcdcd2d84869716ee9139d8944442e20fc125005249b468"} Nov 02 20:41:58.228406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.228240 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" event={"ID":"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1","Type":"ContainerStarted","Data":"41488a702b963da49d3fabbf58667458c01643dabb1a2371239b4e476156b2e6"} Nov 02 20:41:58.343366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.343277 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" podStartSLOduration=2.343263604 podStartE2EDuration="2.343263604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:41:58.337934094 +0000 UTC m=+1646.627590623" watchObservedRunningTime="2026-11-02 20:41:58.343263604 +0000 UTC m=+1646.632920132" Nov 02 20:41:58.803827 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.803804 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:41:58.915769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.915702 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0be871c-832b-4e45-965c-d092b43030eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0be871c-832b-4e45-965c-d092b43030eb-istio-podinfo\") pod \"f0be871c-832b-4e45-965c-d092b43030eb\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " Nov 02 20:41:58.915769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.915747 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-credential-socket\") pod \"f0be871c-832b-4e45-965c-d092b43030eb\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " Nov 02 20:41:58.915962 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.915828 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0be871c-832b-4e45-965c-d092b43030eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0be871c-832b-4e45-965c-d092b43030eb-istiod-ca-cert\") pod \"f0be871c-832b-4e45-965c-d092b43030eb\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " Nov 02 20:41:58.915962 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.915904 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-workload-socket\") pod \"f0be871c-832b-4e45-965c-d092b43030eb\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " Nov 02 20:41:58.915962 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.915950 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0be871c-832b-4e45-965c-d092b43030eb-istio-token\" (UniqueName: \"kubernetes.io/projected/f0be871c-832b-4e45-965c-d092b43030eb-istio-token\") pod \"f0be871c-832b-4e45-965c-d092b43030eb\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " Nov 02 20:41:58.916181 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.916139 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-istio-envoy\") pod \"f0be871c-832b-4e45-965c-d092b43030eb\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " Nov 02 20:41:58.916181 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.916152 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-credential-socket" pod "f0be871c-832b-4e45-965c-d092b43030eb" (UID: "f0be871c-832b-4e45-965c-d092b43030eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:41:58.916296 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.916239 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-workload-certs\") pod \"f0be871c-832b-4e45-965c-d092b43030eb\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " Nov 02 20:41:58.916296 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.916283 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-istio-data\") pod \"f0be871c-832b-4e45-965c-d092b43030eb\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " Nov 02 20:41:58.916403 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.916319 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0be871c-832b-4e45-965c-d092b43030eb-kube-api-access-8vbn4\" (UniqueName: \"kubernetes.io/projected/f0be871c-832b-4e45-965c-d092b43030eb-kube-api-access-8vbn4\") pod \"f0be871c-832b-4e45-965c-d092b43030eb\" (UID: \"f0be871c-832b-4e45-965c-d092b43030eb\") " Nov 02 20:41:58.916678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.916237 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-workload-socket" pod "f0be871c-832b-4e45-965c-d092b43030eb" (UID: "f0be871c-832b-4e45-965c-d092b43030eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:41:58.916678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.916313 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0be871c-832b-4e45-965c-d092b43030eb-istiod-ca-cert" pod "f0be871c-832b-4e45-965c-d092b43030eb" (UID: "f0be871c-832b-4e45-965c-d092b43030eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:41:58.916678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.916560 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-workload-certs" pod "f0be871c-832b-4e45-965c-d092b43030eb" (UID: "f0be871c-832b-4e45-965c-d092b43030eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:41:58.916855 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.916717 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-istio-data" pod "f0be871c-832b-4e45-965c-d092b43030eb" (UID: "f0be871c-832b-4e45-965c-d092b43030eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:41:58.918437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.917332 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0be871c-832b-4e45-965c-d092b43030eb-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:58.918437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.917361 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:58.918437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.917377 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:58.918437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.917393 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:58.918437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.917407 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:58.918795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.918747 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0be871c-832b-4e45-965c-d092b43030eb-istio-podinfo" pod "f0be871c-832b-4e45-965c-d092b43030eb" (UID: "f0be871c-832b-4e45-965c-d092b43030eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:41:58.918795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.918765 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-istio-envoy" pod "f0be871c-832b-4e45-965c-d092b43030eb" (UID: "f0be871c-832b-4e45-965c-d092b43030eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:41:58.919075 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.919048 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0be871c-832b-4e45-965c-d092b43030eb-istio-token" pod "f0be871c-832b-4e45-965c-d092b43030eb" (UID: "f0be871c-832b-4e45-965c-d092b43030eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:41:58.919075 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:58.919120 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0be871c-832b-4e45-965c-d092b43030eb-kube-api-access-8vbn4" pod "f0be871c-832b-4e45-965c-d092b43030eb" (UID: "f0be871c-832b-4e45-965c-d092b43030eb"). InnerVolumeSpecName "kube-api-access-8vbn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:41:59.018955 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:59.018914 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0be871c-832b-4e45-965c-d092b43030eb-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:59.018955 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:59.018947 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0be871c-832b-4e45-965c-d092b43030eb-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:59.019232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:59.018964 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vbn4\" (UniqueName: \"kubernetes.io/projected/f0be871c-832b-4e45-965c-d092b43030eb-kube-api-access-8vbn4\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:59.019232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:59.018980 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0be871c-832b-4e45-965c-d092b43030eb-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:59.241486 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:59.241397 4971 generic.go:356] "Generic (PLEG): container finished" podID="e084e043-2e14-4d59-b82e-6f6b1c8aa3b1" containerID="e81064b4a36e66691fcdcd2d84869716ee9139d8944442e20fc125005249b468" exitCode=0 Nov 02 20:41:59.241890 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:59.241490 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" event={"ID":"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1","Type":"ContainerDied","Data":"e81064b4a36e66691fcdcd2d84869716ee9139d8944442e20fc125005249b468"} Nov 02 20:41:59.243450 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:59.243426 4971 generic.go:356] "Generic (PLEG): container finished" podID="f0be871c-832b-4e45-965c-d092b43030eb" containerID="f56e4dddc8c2f2f840507acd69acac1751371c4a8e581c1eed6d2dade80389fd" exitCode=0 Nov 02 20:41:59.243613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:59.243471 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" Nov 02 20:41:59.243613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:59.243513 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" event={"ID":"f0be871c-832b-4e45-965c-d092b43030eb","Type":"ContainerDied","Data":"f56e4dddc8c2f2f840507acd69acac1751371c4a8e581c1eed6d2dade80389fd"} Nov 02 20:41:59.243613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:59.243574 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75" event={"ID":"f0be871c-832b-4e45-965c-d092b43030eb","Type":"ContainerDied","Data":"eb3362f0afc128396a452395d4a2456390bfd1ff45f4304dcfc17f4a3a170392"} Nov 02 20:41:59.243613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:59.243602 4971 scope.go:122] "RemoveContainer" containerID="f56e4dddc8c2f2f840507acd69acac1751371c4a8e581c1eed6d2dade80389fd" Nov 02 20:41:59.259866 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:59.259848 4971 scope.go:122] "RemoveContainer" containerID="f56e4dddc8c2f2f840507acd69acac1751371c4a8e581c1eed6d2dade80389fd" Nov 02 20:41:59.260201 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:41:59.260181 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56e4dddc8c2f2f840507acd69acac1751371c4a8e581c1eed6d2dade80389fd\": container with ID starting with f56e4dddc8c2f2f840507acd69acac1751371c4a8e581c1eed6d2dade80389fd not found: ID does not exist" containerID="f56e4dddc8c2f2f840507acd69acac1751371c4a8e581c1eed6d2dade80389fd" Nov 02 20:41:59.260265 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:59.260211 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56e4dddc8c2f2f840507acd69acac1751371c4a8e581c1eed6d2dade80389fd"} err="failed to get container status \"f56e4dddc8c2f2f840507acd69acac1751371c4a8e581c1eed6d2dade80389fd\": rpc error: code = NotFound desc = could not find container \"f56e4dddc8c2f2f840507acd69acac1751371c4a8e581c1eed6d2dade80389fd\": container with ID starting with f56e4dddc8c2f2f840507acd69acac1751371c4a8e581c1eed6d2dade80389fd not found: ID does not exist" Nov 02 20:41:59.477916 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:59.477879 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75"] Nov 02 20:41:59.483192 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:59.483169 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fgn75"] Nov 02 20:41:59.868506 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:41:59.868466 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0be871c-832b-4e45-965c-d092b43030eb" path="/var/lib/kubelet/pods/f0be871c-832b-4e45-965c-d092b43030eb/volumes" Nov 02 20:42:00.436935 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:00.436877 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" Nov 02 20:42:00.539544 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:00.539481 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-kube-api-access-x42wx\" (UniqueName: \"kubernetes.io/projected/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-kube-api-access-x42wx\") pod \"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1\" (UID: \"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1\") " Nov 02 20:42:00.539544 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:00.539536 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-maas-api-openshift-ca\") pod \"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1\" (UID: \"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1\") " Nov 02 20:42:00.539701 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:00.539666 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-maas-api-xks-ca\") pod \"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1\" (UID: \"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1\") " Nov 02 20:42:00.539883 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:00.539860 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-maas-api-openshift-ca" pod "e084e043-2e14-4d59-b82e-6f6b1c8aa3b1" (UID: "e084e043-2e14-4d59-b82e-6f6b1c8aa3b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:42:00.540197 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:00.540171 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:00.541773 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:00.541746 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-kube-api-access-x42wx" pod "e084e043-2e14-4d59-b82e-6f6b1c8aa3b1" (UID: "e084e043-2e14-4d59-b82e-6f6b1c8aa3b1"). InnerVolumeSpecName "kube-api-access-x42wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:00.541884 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:00.541864 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-maas-api-xks-ca" pod "e084e043-2e14-4d59-b82e-6f6b1c8aa3b1" (UID: "e084e043-2e14-4d59-b82e-6f6b1c8aa3b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:00.642026 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:00.641990 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:00.642214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:00.642039 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x42wx\" (UniqueName: \"kubernetes.io/projected/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1-kube-api-access-x42wx\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:00.727735 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:00.727699 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:01.053672 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:01.053576 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9"] Nov 02 20:42:01.062022 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:01.061994 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19af21aa-7hvp9"] Nov 02 20:42:01.412621 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:01.412577 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:01.416097 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:01.416050 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" event={"ID":"e084e043-2e14-4d59-b82e-6f6b1c8aa3b1","Type":"ContainerDied","Data":"41488a702b963da49d3fabbf58667458c01643dabb1a2371239b4e476156b2e6"} Nov 02 20:42:01.416707 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:01.416158 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41488a702b963da49d3fabbf58667458c01643dabb1a2371239b4e476156b2e6" Nov 02 20:42:01.416707 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:01.416230 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh" Nov 02 20:42:01.764498 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:01.764389 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:01.872828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:01.872796 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e0a7df-c53e-4fbd-8b6e-90d2adb023c1" path="/var/lib/kubelet/pods/39e0a7df-c53e-4fbd-8b6e-90d2adb023c1/volumes" Nov 02 20:42:02.069492 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:02.069381 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl"] Nov 02 20:42:02.076100 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:02.076057 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b84cc4-nfvnl"] Nov 02 20:42:03.343192 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:42:03.343148 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:42:03.759650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:03.759614 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:03.872690 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:03.872648 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e3c0362-fdcd-4801-9634-657f6ad5dcae" path="/var/lib/kubelet/pods/3e3c0362-fdcd-4801-9634-657f6ad5dcae/volumes" Nov 02 20:42:06.263520 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:06.263479 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:06.571802 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:06.571728 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:07.048244 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:07.048211 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d"] Nov 02 20:42:07.094976 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:07.094945 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000c7454-66r7d"] Nov 02 20:42:07.872133 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:07.872098 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f72fa1-0c92-475d-9231-0c4aeac59f50" path="/var/lib/kubelet/pods/28f72fa1-0c92-475d-9231-0c4aeac59f50/volumes" Nov 02 20:42:08.975585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:08.975549 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:09.249943 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:09.249862 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:09.496463 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:09.496207 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:12.109231 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:12.109143 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb"] Nov 02 20:42:12.126204 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:12.126160 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45639c6-lbwpb"] Nov 02 20:42:13.051528 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:13.051495 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv"] Nov 02 20:42:13.097960 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:13.097926 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e232924-4glqv"] Nov 02 20:42:13.511120 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:13.511071 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb"] Nov 02 20:42:13.877687 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:13.877647 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8abc64d-820c-4e63-b288-e174393765dc" path="/var/lib/kubelet/pods/b8abc64d-820c-4e63-b288-e174393765dc/volumes" Nov 02 20:42:13.878602 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:13.878574 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5c9309-85f8-4d06-9ede-25224452834d" path="/var/lib/kubelet/pods/ef5c9309-85f8-4d06-9ede-25224452834d/volumes" Nov 02 20:42:14.098639 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.098599 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:14.343600 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.343520 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:14.599565 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.599528 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:14.748067 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.748016 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6"] Nov 02 20:42:14.748995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.748970 4971 cpu_manager.go:414] "containerMap: removing container" podUID="e084e043-2e14-4d59-b82e-6f6b1c8aa3b1" containerName="revoke-keys" Nov 02 20:42:14.748995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.748993 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e084e043-2e14-4d59-b82e-6f6b1c8aa3b1" containerName="revoke-keys" Nov 02 20:42:14.749182 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.749005 4971 cpu_manager.go:414] "containerMap: removing container" podUID="f0be871c-832b-4e45-965c-d092b43030eb" containerName="istio-proxy" Nov 02 20:42:14.749182 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.749012 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0be871c-832b-4e45-965c-d092b43030eb" containerName="istio-proxy" Nov 02 20:42:14.749270 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.749254 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="e084e043-2e14-4d59-b82e-6f6b1c8aa3b1" containerName="revoke-keys" Nov 02 20:42:14.749306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.749275 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0be871c-832b-4e45-965c-d092b43030eb" containerName="istio-proxy" Nov 02 20:42:14.754577 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.754558 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" Nov 02 20:42:14.804410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.804376 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:42:14.811041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.811010 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6"] Nov 02 20:42:14.849685 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.849595 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:42:14.920514 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.920483 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg49j\" (UniqueName: \"kubernetes.io/projected/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-kube-api-access-xg49j\") pod \"maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6\" (UID: \"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" Nov 02 20:42:14.920692 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.920628 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6\" (UID: \"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" Nov 02 20:42:14.920798 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:14.920773 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6\" (UID: \"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" Nov 02 20:42:15.022257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:15.022221 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6\" (UID: \"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" Nov 02 20:42:15.022505 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:15.022490 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg49j\" (UniqueName: \"kubernetes.io/projected/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-kube-api-access-xg49j\") pod \"maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6\" (UID: \"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" Nov 02 20:42:15.022604 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:15.022583 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6\" (UID: \"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" Nov 02 20:42:15.023159 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:15.023128 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6\" (UID: \"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" Nov 02 20:42:15.025372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:15.025348 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6\" (UID: \"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" Nov 02 20:42:15.106161 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:15.096166 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg49j\" (UniqueName: \"kubernetes.io/projected/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-kube-api-access-xg49j\") pod \"maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6\" (UID: \"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" Nov 02 20:42:15.380191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:15.376370 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" Nov 02 20:42:15.844019 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:42:15.843978 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e344cf4_35bb_4c56_b3d1_a4ad1409b2c4.slice/crio-43fe260a692698bbeb0193935d566a8420d0eac5dc81bddf8973d0ce1bee6f44 WatchSource:0}: Error finding container 43fe260a692698bbeb0193935d566a8420d0eac5dc81bddf8973d0ce1bee6f44: Status 404 returned error can't find the container with id 43fe260a692698bbeb0193935d566a8420d0eac5dc81bddf8973d0ce1bee6f44 Nov 02 20:42:15.845687 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:15.845659 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6"] Nov 02 20:42:16.061363 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:16.061336 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4"] Nov 02 20:42:16.069139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:16.069108 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fbf498-6hhd4"] Nov 02 20:42:16.646510 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:16.646467 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" event={"ID":"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4","Type":"ContainerStarted","Data":"6bd25a762c899fc3fb423ba4a3caa2cfb1e6af565796b2c41cde4e91dd2707b1"} Nov 02 20:42:16.646685 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:16.646537 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" event={"ID":"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4","Type":"ContainerStarted","Data":"43fe260a692698bbeb0193935d566a8420d0eac5dc81bddf8973d0ce1bee6f44"} Nov 02 20:42:16.759489 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:16.759432 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" podStartSLOduration=2.75941691 podStartE2EDuration="2.75941691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:42:16.755355828 +0000 UTC m=+1665.045012360" watchObservedRunningTime="2026-11-02 20:42:16.75941691 +0000 UTC m=+1665.049073438" Nov 02 20:42:17.050415 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:17.050380 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx"] Nov 02 20:42:17.055210 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:17.055185 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16247d79-tvpfx"] Nov 02 20:42:17.647107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:17.646126 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:17.732836 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:17.732801 4971 generic.go:356] "Generic (PLEG): container finished" podID="4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4" containerID="6bd25a762c899fc3fb423ba4a3caa2cfb1e6af565796b2c41cde4e91dd2707b1" exitCode=0 Nov 02 20:42:17.733302 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:17.733272 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" event={"ID":"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4","Type":"ContainerDied","Data":"6bd25a762c899fc3fb423ba4a3caa2cfb1e6af565796b2c41cde4e91dd2707b1"} Nov 02 20:42:17.885447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:17.885403 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a" path="/var/lib/kubelet/pods/240d9ecb-01ae-4e69-87c2-1ac5bdf6ba9a/volumes" Nov 02 20:42:17.886180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:17.886096 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d887a152-fb99-48f3-b702-779ce22c4e59" path="/var/lib/kubelet/pods/d887a152-fb99-48f3-b702-779ce22c4e59/volumes" Nov 02 20:42:17.892311 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:17.892285 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:18.511759 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:18.511719 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" podUID="4d35c028-a868-46ca-8be5-6cd3023d724c" containerName="payload-processing" containerID="cri-o://5d7b0ad88249ddf9247eb9fd4f4251fe9f23e1dea075c1d0d0deb05b473686d5" gracePeriod=25 Nov 02 20:42:18.765408 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:18.765308 4971 generic.go:356] "Generic (PLEG): container finished" podID="4d35c028-a868-46ca-8be5-6cd3023d724c" containerID="5d7b0ad88249ddf9247eb9fd4f4251fe9f23e1dea075c1d0d0deb05b473686d5" exitCode=0 Nov 02 20:42:18.765768 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:18.765722 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" event={"ID":"4d35c028-a868-46ca-8be5-6cd3023d724c","Type":"ContainerDied","Data":"5d7b0ad88249ddf9247eb9fd4f4251fe9f23e1dea075c1d0d0deb05b473686d5"} Nov 02 20:42:18.829336 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:18.829306 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" Nov 02 20:42:18.952797 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:18.952770 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d35c028-a868-46ca-8be5-6cd3023d724c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d35c028-a868-46ca-8be5-6cd3023d724c-plugins-config-volume\") pod \"4d35c028-a868-46ca-8be5-6cd3023d724c\" (UID: \"4d35c028-a868-46ca-8be5-6cd3023d724c\") " Nov 02 20:42:18.952953 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:18.952843 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d35c028-a868-46ca-8be5-6cd3023d724c-kube-api-access-7qr5l\" (UniqueName: \"kubernetes.io/projected/4d35c028-a868-46ca-8be5-6cd3023d724c-kube-api-access-7qr5l\") pod \"4d35c028-a868-46ca-8be5-6cd3023d724c\" (UID: \"4d35c028-a868-46ca-8be5-6cd3023d724c\") " Nov 02 20:42:18.953227 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:18.953200 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d35c028-a868-46ca-8be5-6cd3023d724c-plugins-config-volume" pod "4d35c028-a868-46ca-8be5-6cd3023d724c" (UID: "4d35c028-a868-46ca-8be5-6cd3023d724c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:42:18.953595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:18.953571 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d35c028-a868-46ca-8be5-6cd3023d724c-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:18.955665 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:18.955634 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d35c028-a868-46ca-8be5-6cd3023d724c-kube-api-access-7qr5l" pod "4d35c028-a868-46ca-8be5-6cd3023d724c" (UID: "4d35c028-a868-46ca-8be5-6cd3023d724c"). InnerVolumeSpecName "kube-api-access-7qr5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:18.970591 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:18.970558 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" Nov 02 20:42:19.054359 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.054281 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-maas-api-openshift-ca\") pod \"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4\" (UID: \"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4\") " Nov 02 20:42:19.054359 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.054350 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-maas-api-xks-ca\") pod \"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4\" (UID: \"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4\") " Nov 02 20:42:19.054544 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.054451 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-kube-api-access-xg49j\" (UniqueName: \"kubernetes.io/projected/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-kube-api-access-xg49j\") pod \"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4\" (UID: \"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4\") " Nov 02 20:42:19.054727 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.054700 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-maas-api-openshift-ca" pod "4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4" (UID: "4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:42:19.055267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.055237 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qr5l\" (UniqueName: \"kubernetes.io/projected/4d35c028-a868-46ca-8be5-6cd3023d724c-kube-api-access-7qr5l\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:19.055382 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.055273 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:19.056629 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.056602 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-maas-api-xks-ca" pod "4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4" (UID: "4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:19.056712 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.056680 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-kube-api-access-xg49j" pod "4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4" (UID: "4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4"). InnerVolumeSpecName "kube-api-access-xg49j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:19.157287 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.157255 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg49j\" (UniqueName: \"kubernetes.io/projected/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-kube-api-access-xg49j\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:19.157287 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.157284 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:19.775809 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.775775 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" Nov 02 20:42:19.775809 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.775793 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb" event={"ID":"4d35c028-a868-46ca-8be5-6cd3023d724c","Type":"ContainerDied","Data":"07ac26eb55965ff7106b13560e5d79c771c55ac4ed9230a231e8607574428ff5"} Nov 02 20:42:19.776404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.775860 4971 scope.go:122] "RemoveContainer" containerID="5d7b0ad88249ddf9247eb9fd4f4251fe9f23e1dea075c1d0d0deb05b473686d5" Nov 02 20:42:19.777644 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.777626 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" Nov 02 20:42:19.777817 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.777788 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6" event={"ID":"4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4","Type":"ContainerDied","Data":"43fe260a692698bbeb0193935d566a8420d0eac5dc81bddf8973d0ce1bee6f44"} Nov 02 20:42:19.777914 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.777847 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43fe260a692698bbeb0193935d566a8420d0eac5dc81bddf8973d0ce1bee6f44" Nov 02 20:42:19.933283 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.933251 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb"] Nov 02 20:42:19.980613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:19.980586 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df17143f-f5d9549d-df4wb"] Nov 02 20:42:20.129332 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:20.129297 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:21.049376 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:21.049347 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc"] Nov 02 20:42:21.096066 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:21.096037 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-077f127e-schzc"] Nov 02 20:42:21.870228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:21.870196 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d35c028-a868-46ca-8be5-6cd3023d724c" path="/var/lib/kubelet/pods/4d35c028-a868-46ca-8be5-6cd3023d724c/volumes" Nov 02 20:42:21.872427 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:21.872396 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3ecf0f-5677-4afa-884b-a48f7ccd99f5" path="/var/lib/kubelet/pods/ad3ecf0f-5677-4afa-884b-a48f7ccd99f5/volumes" Nov 02 20:42:22.048141 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:22.048069 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg"] Nov 02 20:42:22.094458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:22.094430 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7bd60a1-wbpfg"] Nov 02 20:42:23.870493 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:23.870452 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c4cf0f5-1c2f-411b-b52c-441569c8e62e" path="/var/lib/kubelet/pods/5c4cf0f5-1c2f-411b-b52c-441569c8e62e/volumes" Nov 02 20:42:25.323325 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:25.323293 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:28.050069 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:28.050027 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v"] Nov 02 20:42:28.096350 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:28.096321 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-694b51ab-lhm4v"] Nov 02 20:42:29.047798 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:29.047760 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh"] Nov 02 20:42:29.094031 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:29.093996 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c393931-5fmzh"] Nov 02 20:42:29.871526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:29.871495 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2910c32a-6f07-47fd-8b35-bade9e68d34d" path="/var/lib/kubelet/pods/2910c32a-6f07-47fd-8b35-bade9e68d34d/volumes" Nov 02 20:42:29.872394 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:29.872156 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a173d1e4-942e-4773-b9a4-ae854a61072d" path="/var/lib/kubelet/pods/a173d1e4-942e-4773-b9a4-ae854a61072d/volumes" Nov 02 20:42:30.015442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:30.015406 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:30.258818 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:30.258735 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:30.502107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:30.502060 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:31.327931 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:31.327894 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:31.578342 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:31.578264 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:32.505752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:32.505707 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:32.756680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:32.756574 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:33.340797 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:42:33.340754 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:42:35.084542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:35.084503 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:38.050789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:38.050751 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj"] Nov 02 20:42:38.097031 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:38.097003 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c2eff6-t6zbj"] Nov 02 20:42:38.248122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:38.248077 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:39.870590 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:39.870547 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4d09ac-83dd-4b68-8784-6b5b959498a2" path="/var/lib/kubelet/pods/db4d09ac-83dd-4b68-8784-6b5b959498a2/volumes" Nov 02 20:42:40.113536 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:40.113495 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:41.205802 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:41.205761 4971 scope.go:122] "RemoveContainer" containerID="ef21fd106740179d2952005d98f5a043d0723d8290e0eb6289bbb673c893fe0f" Nov 02 20:42:41.217951 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:41.217912 4971 scope.go:122] "RemoveContainer" containerID="d4e436bdf4c3a97104877ca0e7b511379a9d157f2453350f6b7e0ecc8d802a15" Nov 02 20:42:41.228369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:41.228348 4971 scope.go:122] "RemoveContainer" containerID="ff20cd1e2d0341ef9deccdaf4c018ac9c5d4a215f1aa784572d8dc09673b764d" Nov 02 20:42:41.238624 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:41.238605 4971 scope.go:122] "RemoveContainer" containerID="b2009b532549c3410193e9776302f8c593fa0c889960e778a293a096d3fb21b8" Nov 02 20:42:41.248336 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:41.248315 4971 scope.go:122] "RemoveContainer" containerID="c5cfc23caa3de755ce27b7e050f661057ce0e7171a01ad2309435d7c66ffeefa" Nov 02 20:42:41.261203 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:41.261180 4971 scope.go:122] "RemoveContainer" containerID="2c8b904c1756060e8fcd3afcca3dbb505756cecf2e23e0ed7a8cf41bf3b8d09b" Nov 02 20:42:41.270938 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:41.270906 4971 scope.go:122] "RemoveContainer" containerID="87dc388f5c34f8eb70cb644502f4067cf4ab495f382a421354fd463623617228" Nov 02 20:42:41.282140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:41.282121 4971 scope.go:122] "RemoveContainer" containerID="89cd33741715d167584682aa4ed68da5f50bbf5362cd16b2de68456388720737" Nov 02 20:42:41.291562 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:41.291539 4971 scope.go:122] "RemoveContainer" containerID="fb0b2b9c8681456aa8912f30e008b7a0e9dff9a3c72311cc1fbbca62ee45d61e" Nov 02 20:42:41.301345 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:41.301329 4971 scope.go:122] "RemoveContainer" containerID="0d1563d72abf61e6a47babd707db6d8c2048d7e440f17fced6a8be58d7eeafa6" Nov 02 20:42:44.930596 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:44.930498 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:47.049401 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:47.049371 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd"] Nov 02 20:42:47.094673 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:47.094646 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e623aad5-lwrqd"] Nov 02 20:42:47.868667 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:47.868637 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe" path="/var/lib/kubelet/pods/2cd2a50d-64c2-4ace-bc50-d3ac3d8c70fe/volumes" Nov 02 20:42:49.166405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:49.166357 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:49.620251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:49.620217 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:50.072193 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:50.072059 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55"] Nov 02 20:42:50.112434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:50.112403 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64df16ca-8nl55"] Nov 02 20:42:50.792592 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:50.792549 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:51.144932 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:51.144891 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:51.237553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:51.237518 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh"] Nov 02 20:42:51.283837 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:51.283810 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df17143f-nckbh"] Nov 02 20:42:51.417092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:51.417004 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:51.869487 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:51.869454 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e084e043-2e14-4d59-b82e-6f6b1c8aa3b1" path="/var/lib/kubelet/pods/e084e043-2e14-4d59-b82e-6f6b1c8aa3b1/volumes" Nov 02 20:42:51.870076 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:51.870055 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd952597-9010-4a13-83b6-f178f7942885" path="/var/lib/kubelet/pods/fd952597-9010-4a13-83b6-f178f7942885/volumes" Nov 02 20:42:55.306981 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:55.306941 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:42:56.421094 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:42:56.421056 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:00.051562 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:00.051525 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28"] Nov 02 20:43:00.097804 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:00.097777 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d28439-vhd28"] Nov 02 20:43:00.780374 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:00.780336 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:01.870431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:01.870385 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5989a61f-f5ac-4ab2-8888-9f6568b02ee9" path="/var/lib/kubelet/pods/5989a61f-f5ac-4ab2-8888-9f6568b02ee9/volumes" Nov 02 20:43:02.009064 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:02.009030 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz"] Nov 02 20:43:02.009381 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:02.009345 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" podUID="a50c847f-3e10-42a9-8ca1-1e285db2af51" containerName="istio-proxy" containerID="cri-o://ca9a2dc23966395329b8e7ab0951697da61a862074071bb33b161e21d028ddeb" gracePeriod=30 Nov 02 20:43:02.112971 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:02.112915 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:02.517346 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:02.517316 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:02.680372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:02.680321 4971 patch_prober.go:29] interesting pod/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:43:02.680724 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:02.680691 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" podUID="a50c847f-3e10-42a9-8ca1-1e285db2af51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:43:02.765918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:02.765887 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:03.070757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:03.070721 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6"] Nov 02 20:43:03.117017 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:03.116988 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23d588ad-8kqx6"] Nov 02 20:43:03.117182 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:03.117037 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:03.342561 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:43:03.342475 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:43:03.365723 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:03.365698 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:03.648817 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:03.648619 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:03.914162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:03.913771 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:03.930128 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:03.930056 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4" path="/var/lib/kubelet/pods/4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4/volumes" Nov 02 20:43:04.173769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:04.173678 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:07.306193 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.306158 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:43:07.362399 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.362360 4971 generic.go:356] "Generic (PLEG): container finished" podID="a50c847f-3e10-42a9-8ca1-1e285db2af51" containerID="ca9a2dc23966395329b8e7ab0951697da61a862074071bb33b161e21d028ddeb" exitCode=0 Nov 02 20:43:07.362637 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.362617 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" event={"ID":"a50c847f-3e10-42a9-8ca1-1e285db2af51","Type":"ContainerDied","Data":"ca9a2dc23966395329b8e7ab0951697da61a862074071bb33b161e21d028ddeb"} Nov 02 20:43:07.362726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.362708 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" event={"ID":"a50c847f-3e10-42a9-8ca1-1e285db2af51","Type":"ContainerDied","Data":"3f390780834bc4e2003a5137c3aba7ec3de1bb48d604a97508b67fbd73a48b8c"} Nov 02 20:43:07.362789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.362748 4971 scope.go:122] "RemoveContainer" containerID="ca9a2dc23966395329b8e7ab0951697da61a862074071bb33b161e21d028ddeb" Nov 02 20:43:07.362921 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.362905 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz" Nov 02 20:43:07.386925 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.386902 4971 scope.go:122] "RemoveContainer" containerID="cd843bc1894ec2b4d1b846cf3045d9c6371356d9a2e5abccf93ffd69a7e6ec46" Nov 02 20:43:07.401695 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.401666 4971 scope.go:122] "RemoveContainer" containerID="ca9a2dc23966395329b8e7ab0951697da61a862074071bb33b161e21d028ddeb" Nov 02 20:43:07.402157 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:43:07.402135 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9a2dc23966395329b8e7ab0951697da61a862074071bb33b161e21d028ddeb\": container with ID starting with ca9a2dc23966395329b8e7ab0951697da61a862074071bb33b161e21d028ddeb not found: ID does not exist" containerID="ca9a2dc23966395329b8e7ab0951697da61a862074071bb33b161e21d028ddeb" Nov 02 20:43:07.402242 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.402169 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9a2dc23966395329b8e7ab0951697da61a862074071bb33b161e21d028ddeb"} err="failed to get container status \"ca9a2dc23966395329b8e7ab0951697da61a862074071bb33b161e21d028ddeb\": rpc error: code = NotFound desc = could not find container \"ca9a2dc23966395329b8e7ab0951697da61a862074071bb33b161e21d028ddeb\": container with ID starting with ca9a2dc23966395329b8e7ab0951697da61a862074071bb33b161e21d028ddeb not found: ID does not exist" Nov 02 20:43:07.402242 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.402202 4971 scope.go:122] "RemoveContainer" containerID="cd843bc1894ec2b4d1b846cf3045d9c6371356d9a2e5abccf93ffd69a7e6ec46" Nov 02 20:43:07.402551 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:43:07.402527 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd843bc1894ec2b4d1b846cf3045d9c6371356d9a2e5abccf93ffd69a7e6ec46\": container with ID starting with cd843bc1894ec2b4d1b846cf3045d9c6371356d9a2e5abccf93ffd69a7e6ec46 not found: ID does not exist" containerID="cd843bc1894ec2b4d1b846cf3045d9c6371356d9a2e5abccf93ffd69a7e6ec46" Nov 02 20:43:07.402623 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.402559 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd843bc1894ec2b4d1b846cf3045d9c6371356d9a2e5abccf93ffd69a7e6ec46"} err="failed to get container status \"cd843bc1894ec2b4d1b846cf3045d9c6371356d9a2e5abccf93ffd69a7e6ec46\": rpc error: code = NotFound desc = could not find container \"cd843bc1894ec2b4d1b846cf3045d9c6371356d9a2e5abccf93ffd69a7e6ec46\": container with ID starting with cd843bc1894ec2b4d1b846cf3045d9c6371356d9a2e5abccf93ffd69a7e6ec46 not found: ID does not exist" Nov 02 20:43:07.460505 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.460480 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-podinfo\") pod \"a50c847f-3e10-42a9-8ca1-1e285db2af51\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " Nov 02 20:43:07.460661 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.460528 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-token\" (UniqueName: \"kubernetes.io/projected/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-token\") pod \"a50c847f-3e10-42a9-8ca1-1e285db2af51\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " Nov 02 20:43:07.460661 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.460605 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a50c847f-3e10-42a9-8ca1-1e285db2af51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a50c847f-3e10-42a9-8ca1-1e285db2af51-istiod-ca-cert\") pod \"a50c847f-3e10-42a9-8ca1-1e285db2af51\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " Nov 02 20:43:07.460661 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.460644 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50c847f-3e10-42a9-8ca1-1e285db2af51-kube-api-access-bblnq\" (UniqueName: \"kubernetes.io/projected/a50c847f-3e10-42a9-8ca1-1e285db2af51-kube-api-access-bblnq\") pod \"a50c847f-3e10-42a9-8ca1-1e285db2af51\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " Nov 02 20:43:07.460822 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.460690 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-data\") pod \"a50c847f-3e10-42a9-8ca1-1e285db2af51\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " Nov 02 20:43:07.460822 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.460758 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-envoy\") pod \"a50c847f-3e10-42a9-8ca1-1e285db2af51\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " Nov 02 20:43:07.460932 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.460826 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-workload-certs\") pod \"a50c847f-3e10-42a9-8ca1-1e285db2af51\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " Nov 02 20:43:07.460932 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.460867 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-workload-socket\") pod \"a50c847f-3e10-42a9-8ca1-1e285db2af51\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " Nov 02 20:43:07.461035 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.461009 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-credential-socket\") pod \"a50c847f-3e10-42a9-8ca1-1e285db2af51\" (UID: \"a50c847f-3e10-42a9-8ca1-1e285db2af51\") " Nov 02 20:43:07.461876 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.461702 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-workload-certs" pod "a50c847f-3e10-42a9-8ca1-1e285db2af51" (UID: "a50c847f-3e10-42a9-8ca1-1e285db2af51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:07.462021 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.461889 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-credential-socket" pod "a50c847f-3e10-42a9-8ca1-1e285db2af51" (UID: "a50c847f-3e10-42a9-8ca1-1e285db2af51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:07.462201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.462058 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-workload-socket" pod "a50c847f-3e10-42a9-8ca1-1e285db2af51" (UID: "a50c847f-3e10-42a9-8ca1-1e285db2af51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:07.462549 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.462265 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-data" pod "a50c847f-3e10-42a9-8ca1-1e285db2af51" (UID: "a50c847f-3e10-42a9-8ca1-1e285db2af51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:07.462688 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.462658 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.462791 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.462768 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50c847f-3e10-42a9-8ca1-1e285db2af51-istiod-ca-cert" pod "a50c847f-3e10-42a9-8ca1-1e285db2af51" (UID: "a50c847f-3e10-42a9-8ca1-1e285db2af51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:07.463412 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.463387 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-podinfo" pod "a50c847f-3e10-42a9-8ca1-1e285db2af51" (UID: "a50c847f-3e10-42a9-8ca1-1e285db2af51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:07.463678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.463655 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50c847f-3e10-42a9-8ca1-1e285db2af51-kube-api-access-bblnq" pod "a50c847f-3e10-42a9-8ca1-1e285db2af51" (UID: "a50c847f-3e10-42a9-8ca1-1e285db2af51"). InnerVolumeSpecName "kube-api-access-bblnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:07.464495 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.464468 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-envoy" pod "a50c847f-3e10-42a9-8ca1-1e285db2af51" (UID: "a50c847f-3e10-42a9-8ca1-1e285db2af51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:07.464657 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.464639 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-token" pod "a50c847f-3e10-42a9-8ca1-1e285db2af51" (UID: "a50c847f-3e10-42a9-8ca1-1e285db2af51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:07.564762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.564696 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.564762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.564722 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.564762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.564731 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.564762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.564739 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.564762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.564748 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a50c847f-3e10-42a9-8ca1-1e285db2af51-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.564762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.564757 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bblnq\" (UniqueName: \"kubernetes.io/projected/a50c847f-3e10-42a9-8ca1-1e285db2af51-kube-api-access-bblnq\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.564762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.564766 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.565066 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.564775 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a50c847f-3e10-42a9-8ca1-1e285db2af51-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.782168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.782141 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz"] Nov 02 20:43:07.786497 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.786474 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-df17143f-openshift-default-6b844db99c-wn6mz"] Nov 02 20:43:07.869477 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:07.869440 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a50c847f-3e10-42a9-8ca1-1e285db2af51" path="/var/lib/kubelet/pods/a50c847f-3e10-42a9-8ca1-1e285db2af51/volumes" Nov 02 20:43:08.397652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:08.397436 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624"] Nov 02 20:43:08.398230 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:08.397699 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" podUID="714bd47a-f22d-42ac-9692-b6080d740fc0" containerName="istio-proxy" containerID="cri-o://ebdc4dadcf213cf5563e8e13a1f429ab72fef9af5fad134765aeee8618c2816d" gracePeriod=30 Nov 02 20:43:08.434578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:08.434309 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:08.745676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:08.745575 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:09.014416 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:09.014339 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:09.618743 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:09.618710 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:09.902322 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:09.902241 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:10.107893 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.107864 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l"] Nov 02 20:43:10.109628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.109590 4971 cpu_manager.go:414] "containerMap: removing container" podUID="a50c847f-3e10-42a9-8ca1-1e285db2af51" containerName="istio-proxy" Nov 02 20:43:10.109783 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.109628 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50c847f-3e10-42a9-8ca1-1e285db2af51" containerName="istio-proxy" Nov 02 20:43:10.109846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.109789 4971 cpu_manager.go:414] "containerMap: removing container" podUID="4d35c028-a868-46ca-8be5-6cd3023d724c" containerName="payload-processing" Nov 02 20:43:10.109846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.109799 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d35c028-a868-46ca-8be5-6cd3023d724c" containerName="payload-processing" Nov 02 20:43:10.109846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.109835 4971 cpu_manager.go:414] "containerMap: removing container" podUID="4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4" containerName="revoke-keys" Nov 02 20:43:10.109846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.109842 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4" containerName="revoke-keys" Nov 02 20:43:10.110352 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.110332 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d35c028-a868-46ca-8be5-6cd3023d724c" containerName="payload-processing" Nov 02 20:43:10.110442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.110367 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e344cf4-35bb-4c56-b3d1-a4ad1409b2c4" containerName="revoke-keys" Nov 02 20:43:10.110442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.110388 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50c847f-3e10-42a9-8ca1-1e285db2af51" containerName="istio-proxy" Nov 02 20:43:10.117828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.117792 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" Nov 02 20:43:10.167972 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.167891 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l"] Nov 02 20:43:10.207971 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.207942 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e672784a-3207-4ded-858f-07e0daf07b28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-694cee-js82l\" (UID: \"e672784a-3207-4ded-858f-07e0daf07b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" Nov 02 20:43:10.208144 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.208060 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e672784a-3207-4ded-858f-07e0daf07b28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-694cee-js82l\" (UID: \"e672784a-3207-4ded-858f-07e0daf07b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" Nov 02 20:43:10.208279 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.208239 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:43:10.208339 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.208283 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:43:10.208339 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.208302 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnkd5\" (UniqueName: \"kubernetes.io/projected/e672784a-3207-4ded-858f-07e0daf07b28-kube-api-access-cnkd5\") pod \"maas-api-revoke-keys-e2e-worker-w2-694cee-js82l\" (UID: \"e672784a-3207-4ded-858f-07e0daf07b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" Nov 02 20:43:10.310195 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.310161 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e672784a-3207-4ded-858f-07e0daf07b28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-694cee-js82l\" (UID: \"e672784a-3207-4ded-858f-07e0daf07b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" Nov 02 20:43:10.310365 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.310346 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnkd5\" (UniqueName: \"kubernetes.io/projected/e672784a-3207-4ded-858f-07e0daf07b28-kube-api-access-cnkd5\") pod \"maas-api-revoke-keys-e2e-worker-w2-694cee-js82l\" (UID: \"e672784a-3207-4ded-858f-07e0daf07b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" Nov 02 20:43:10.310516 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.310502 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e672784a-3207-4ded-858f-07e0daf07b28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-694cee-js82l\" (UID: \"e672784a-3207-4ded-858f-07e0daf07b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" Nov 02 20:43:10.310847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.310824 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e672784a-3207-4ded-858f-07e0daf07b28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-694cee-js82l\" (UID: \"e672784a-3207-4ded-858f-07e0daf07b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" Nov 02 20:43:10.312996 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.312971 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e672784a-3207-4ded-858f-07e0daf07b28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-694cee-js82l\" (UID: \"e672784a-3207-4ded-858f-07e0daf07b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" Nov 02 20:43:10.367780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.367752 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnkd5\" (UniqueName: \"kubernetes.io/projected/e672784a-3207-4ded-858f-07e0daf07b28-kube-api-access-cnkd5\") pod \"maas-api-revoke-keys-e2e-worker-w2-694cee-js82l\" (UID: \"e672784a-3207-4ded-858f-07e0daf07b28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" Nov 02 20:43:10.432733 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.432682 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" Nov 02 20:43:10.809281 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:43:10.809246 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode672784a_3207_4ded_858f_07e0daf07b28.slice/crio-4f0e45aafca85b2155c9f7a2b437e0152935917a4174e82267f17cd2536ee94c WatchSource:0}: Error finding container 4f0e45aafca85b2155c9f7a2b437e0152935917a4174e82267f17cd2536ee94c: Status 404 returned error can't find the container with id 4f0e45aafca85b2155c9f7a2b437e0152935917a4174e82267f17cd2536ee94c Nov 02 20:43:10.810018 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:10.809976 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l"] Nov 02 20:43:11.421333 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:11.421249 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" event={"ID":"e672784a-3207-4ded-858f-07e0daf07b28","Type":"ContainerStarted","Data":"7e1d99c5653a85d32887db7c57f038a99500bd3ac23f473dcae9a8f05d8cc150"} Nov 02 20:43:11.421333 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:11.421306 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" event={"ID":"e672784a-3207-4ded-858f-07e0daf07b28","Type":"ContainerStarted","Data":"4f0e45aafca85b2155c9f7a2b437e0152935917a4174e82267f17cd2536ee94c"} Nov 02 20:43:11.535410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:11.535357 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" podStartSLOduration=1.53534318 podStartE2EDuration="1.53534318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:43:11.533009065 +0000 UTC m=+1719.822665592" watchObservedRunningTime="2026-11-02 20:43:11.53534318 +0000 UTC m=+1719.824999740" Nov 02 20:43:12.431652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:12.431617 4971 generic.go:356] "Generic (PLEG): container finished" podID="e672784a-3207-4ded-858f-07e0daf07b28" containerID="7e1d99c5653a85d32887db7c57f038a99500bd3ac23f473dcae9a8f05d8cc150" exitCode=0 Nov 02 20:43:12.432490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:12.431671 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" event={"ID":"e672784a-3207-4ded-858f-07e0daf07b28","Type":"ContainerDied","Data":"7e1d99c5653a85d32887db7c57f038a99500bd3ac23f473dcae9a8f05d8cc150"} Nov 02 20:43:13.643631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.643278 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" Nov 02 20:43:13.714665 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.714637 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:43:13.764290 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.764258 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-credential-socket\") pod \"714bd47a-f22d-42ac-9692-b6080d740fc0\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " Nov 02 20:43:13.764457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.764441 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e672784a-3207-4ded-858f-07e0daf07b28-kube-api-access-cnkd5\" (UniqueName: \"kubernetes.io/projected/e672784a-3207-4ded-858f-07e0daf07b28-kube-api-access-cnkd5\") pod \"e672784a-3207-4ded-858f-07e0daf07b28\" (UID: \"e672784a-3207-4ded-858f-07e0daf07b28\") " Nov 02 20:43:13.764499 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.764486 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e672784a-3207-4ded-858f-07e0daf07b28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e672784a-3207-4ded-858f-07e0daf07b28-maas-api-xks-ca\") pod \"e672784a-3207-4ded-858f-07e0daf07b28\" (UID: \"e672784a-3207-4ded-858f-07e0daf07b28\") " Nov 02 20:43:13.764570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.764558 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-workload-certs\") pod \"714bd47a-f22d-42ac-9692-b6080d740fc0\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " Nov 02 20:43:13.764611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.764555 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-credential-socket" pod "714bd47a-f22d-42ac-9692-b6080d740fc0" (UID: "714bd47a-f22d-42ac-9692-b6080d740fc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:13.764611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.764584 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714bd47a-f22d-42ac-9692-b6080d740fc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714bd47a-f22d-42ac-9692-b6080d740fc0-istiod-ca-cert\") pod \"714bd47a-f22d-42ac-9692-b6080d740fc0\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " Nov 02 20:43:13.764894 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.764870 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714bd47a-f22d-42ac-9692-b6080d740fc0-istiod-ca-cert" pod "714bd47a-f22d-42ac-9692-b6080d740fc0" (UID: "714bd47a-f22d-42ac-9692-b6080d740fc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:13.765046 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.765027 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714bd47a-f22d-42ac-9692-b6080d740fc0-kube-api-access-75bq9\" (UniqueName: \"kubernetes.io/projected/714bd47a-f22d-42ac-9692-b6080d740fc0-kube-api-access-75bq9\") pod \"714bd47a-f22d-42ac-9692-b6080d740fc0\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " Nov 02 20:43:13.765174 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.765059 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-workload-certs" pod "714bd47a-f22d-42ac-9692-b6080d740fc0" (UID: "714bd47a-f22d-42ac-9692-b6080d740fc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:13.765174 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.765069 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-workload-socket\") pod \"714bd47a-f22d-42ac-9692-b6080d740fc0\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " Nov 02 20:43:13.765174 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.765124 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-data\") pod \"714bd47a-f22d-42ac-9692-b6080d740fc0\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " Nov 02 20:43:13.765355 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.765182 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-token\" (UniqueName: \"kubernetes.io/projected/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-token\") pod \"714bd47a-f22d-42ac-9692-b6080d740fc0\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " Nov 02 20:43:13.765355 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.765211 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-envoy\") pod \"714bd47a-f22d-42ac-9692-b6080d740fc0\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " Nov 02 20:43:13.765355 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.765250 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e672784a-3207-4ded-858f-07e0daf07b28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e672784a-3207-4ded-858f-07e0daf07b28-maas-api-openshift-ca\") pod \"e672784a-3207-4ded-858f-07e0daf07b28\" (UID: \"e672784a-3207-4ded-858f-07e0daf07b28\") " Nov 02 20:43:13.765355 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.765285 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-podinfo\") pod \"714bd47a-f22d-42ac-9692-b6080d740fc0\" (UID: \"714bd47a-f22d-42ac-9692-b6080d740fc0\") " Nov 02 20:43:13.765572 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.765409 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-workload-socket" pod "714bd47a-f22d-42ac-9692-b6080d740fc0" (UID: "714bd47a-f22d-42ac-9692-b6080d740fc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:13.766107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.766056 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-data" pod "714bd47a-f22d-42ac-9692-b6080d740fc0" (UID: "714bd47a-f22d-42ac-9692-b6080d740fc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:13.766242 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.766216 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:13.766242 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.766240 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714bd47a-f22d-42ac-9692-b6080d740fc0-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:13.766374 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.766255 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:13.766374 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.766269 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:13.766883 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.766860 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e672784a-3207-4ded-858f-07e0daf07b28-maas-api-openshift-ca" pod "e672784a-3207-4ded-858f-07e0daf07b28" (UID: "e672784a-3207-4ded-858f-07e0daf07b28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:13.767869 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.767839 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e672784a-3207-4ded-858f-07e0daf07b28-maas-api-xks-ca" pod "e672784a-3207-4ded-858f-07e0daf07b28" (UID: "e672784a-3207-4ded-858f-07e0daf07b28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:13.767974 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.767910 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714bd47a-f22d-42ac-9692-b6080d740fc0-kube-api-access-75bq9" pod "714bd47a-f22d-42ac-9692-b6080d740fc0" (UID: "714bd47a-f22d-42ac-9692-b6080d740fc0"). InnerVolumeSpecName "kube-api-access-75bq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:13.768526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.768472 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e672784a-3207-4ded-858f-07e0daf07b28-kube-api-access-cnkd5" pod "e672784a-3207-4ded-858f-07e0daf07b28" (UID: "e672784a-3207-4ded-858f-07e0daf07b28"). InnerVolumeSpecName "kube-api-access-cnkd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:13.768526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.768475 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-envoy" pod "714bd47a-f22d-42ac-9692-b6080d740fc0" (UID: "714bd47a-f22d-42ac-9692-b6080d740fc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:13.768819 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.768788 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-podinfo" pod "714bd47a-f22d-42ac-9692-b6080d740fc0" (UID: "714bd47a-f22d-42ac-9692-b6080d740fc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:13.769348 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.769322 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-token" pod "714bd47a-f22d-42ac-9692-b6080d740fc0" (UID: "714bd47a-f22d-42ac-9692-b6080d740fc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:13.850933 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.850896 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:13.868652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.868471 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnkd5\" (UniqueName: \"kubernetes.io/projected/e672784a-3207-4ded-858f-07e0daf07b28-kube-api-access-cnkd5\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:13.868652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.868514 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e672784a-3207-4ded-858f-07e0daf07b28-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:13.868652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.868530 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75bq9\" (UniqueName: \"kubernetes.io/projected/714bd47a-f22d-42ac-9692-b6080d740fc0-kube-api-access-75bq9\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:13.868652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.868544 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:13.868652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.868562 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:13.868652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.868578 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:13.868652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.868592 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e672784a-3207-4ded-858f-07e0daf07b28-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:13.868652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:13.868606 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714bd47a-f22d-42ac-9692-b6080d740fc0-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:14.099317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:14.099279 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:14.506350 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:14.506318 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" Nov 02 20:43:14.521105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:14.507398 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l" event={"ID":"e672784a-3207-4ded-858f-07e0daf07b28","Type":"ContainerDied","Data":"4f0e45aafca85b2155c9f7a2b437e0152935917a4174e82267f17cd2536ee94c"} Nov 02 20:43:14.521910 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:14.521808 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f0e45aafca85b2155c9f7a2b437e0152935917a4174e82267f17cd2536ee94c" Nov 02 20:43:14.522127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:14.522102 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" event={"ID":"714bd47a-f22d-42ac-9692-b6080d740fc0","Type":"ContainerDied","Data":"ebdc4dadcf213cf5563e8e13a1f429ab72fef9af5fad134765aeee8618c2816d"} Nov 02 20:43:14.524150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:14.510950 4971 generic.go:356] "Generic (PLEG): container finished" podID="714bd47a-f22d-42ac-9692-b6080d740fc0" containerID="ebdc4dadcf213cf5563e8e13a1f429ab72fef9af5fad134765aeee8618c2816d" exitCode=0 Nov 02 20:43:14.524150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:14.522489 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" event={"ID":"714bd47a-f22d-42ac-9692-b6080d740fc0","Type":"ContainerDied","Data":"ed93e058e9ffffba03f5087c3cce0e7b334f055d1777756bbf7d939b257c22f4"} Nov 02 20:43:14.524150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:14.522365 4971 scope.go:122] "RemoveContainer" containerID="ebdc4dadcf213cf5563e8e13a1f429ab72fef9af5fad134765aeee8618c2816d" Nov 02 20:43:14.524150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:14.511030 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624" Nov 02 20:43:14.542107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:14.542066 4971 scope.go:122] "RemoveContainer" containerID="ebdc4dadcf213cf5563e8e13a1f429ab72fef9af5fad134765aeee8618c2816d" Nov 02 20:43:14.542667 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:43:14.542642 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebdc4dadcf213cf5563e8e13a1f429ab72fef9af5fad134765aeee8618c2816d\": container with ID starting with ebdc4dadcf213cf5563e8e13a1f429ab72fef9af5fad134765aeee8618c2816d not found: ID does not exist" containerID="ebdc4dadcf213cf5563e8e13a1f429ab72fef9af5fad134765aeee8618c2816d" Nov 02 20:43:14.542771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:14.542674 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebdc4dadcf213cf5563e8e13a1f429ab72fef9af5fad134765aeee8618c2816d"} err="failed to get container status \"ebdc4dadcf213cf5563e8e13a1f429ab72fef9af5fad134765aeee8618c2816d\": rpc error: code = NotFound desc = could not find container \"ebdc4dadcf213cf5563e8e13a1f429ab72fef9af5fad134765aeee8618c2816d\": container with ID starting with ebdc4dadcf213cf5563e8e13a1f429ab72fef9af5fad134765aeee8618c2816d not found: ID does not exist" Nov 02 20:43:14.554145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:14.551513 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7"] Nov 02 20:43:14.599656 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:14.599623 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7"] Nov 02 20:43:14.702974 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:14.702943 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624"] Nov 02 20:43:14.707603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:14.707577 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-ph624"] Nov 02 20:43:14.761750 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:14.761685 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:15.868605 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:15.868573 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="714bd47a-f22d-42ac-9692-b6080d740fc0" path="/var/lib/kubelet/pods/714bd47a-f22d-42ac-9692-b6080d740fc0/volumes" Nov 02 20:43:16.578735 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:16.578684 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:17.003207 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.003177 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr"] Nov 02 20:43:17.004063 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.004042 4971 cpu_manager.go:414] "containerMap: removing container" podUID="714bd47a-f22d-42ac-9692-b6080d740fc0" containerName="istio-proxy" Nov 02 20:43:17.004063 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.004063 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="714bd47a-f22d-42ac-9692-b6080d740fc0" containerName="istio-proxy" Nov 02 20:43:17.004225 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.004098 4971 cpu_manager.go:414] "containerMap: removing container" podUID="a50c847f-3e10-42a9-8ca1-1e285db2af51" containerName="istio-proxy" Nov 02 20:43:17.004225 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.004104 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50c847f-3e10-42a9-8ca1-1e285db2af51" containerName="istio-proxy" Nov 02 20:43:17.004225 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.004139 4971 cpu_manager.go:414] "containerMap: removing container" podUID="e672784a-3207-4ded-858f-07e0daf07b28" containerName="revoke-keys" Nov 02 20:43:17.004225 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.004144 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e672784a-3207-4ded-858f-07e0daf07b28" containerName="revoke-keys" Nov 02 20:43:17.004349 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.004244 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50c847f-3e10-42a9-8ca1-1e285db2af51" containerName="istio-proxy" Nov 02 20:43:17.004349 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.004252 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="714bd47a-f22d-42ac-9692-b6080d740fc0" containerName="istio-proxy" Nov 02 20:43:17.004349 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.004266 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="e672784a-3207-4ded-858f-07e0daf07b28" containerName="revoke-keys" Nov 02 20:43:17.009915 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.009894 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.070652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.070615 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-data\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.070910 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.070880 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-podinfo\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.070989 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.070925 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-token\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.070989 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.070953 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkkrn\" (UniqueName: \"kubernetes.io/projected/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-kube-api-access-xkkrn\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.071150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.071047 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-workload-socket\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.071150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.071107 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istiod-ca-cert\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.071276 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.071193 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-credential-socket\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.071276 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.071256 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-workload-certs\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.071380 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.071345 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-envoy\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.075165 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.075135 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr"] Nov 02 20:43:17.108323 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.108298 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-1620dc-openshift-default-dockercfg-647l4\"" Nov 02 20:43:17.224107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.174627 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-credential-socket\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.225000 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.209906 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-credential-socket\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.225235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.225213 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-workload-certs\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.225766 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.225746 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-envoy\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.226506 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.226486 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-data\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.249041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.249011 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-podinfo\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.249705 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.249682 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-token\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.265647 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.265578 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkkrn\" (UniqueName: \"kubernetes.io/projected/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-kube-api-access-xkkrn\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.265905 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.265886 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-workload-socket\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.287273 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.287179 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istiod-ca-cert\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.287431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.225630 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-workload-certs\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.287431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.265996 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-podinfo\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.287431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.247369 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-data\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.287431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.248855 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-envoy\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.287431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.266506 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-workload-socket\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.287741 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.246864 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:17.290137 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.288130 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istiod-ca-cert\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.342109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.339686 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-token\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.342313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.342130 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkkrn\" (UniqueName: \"kubernetes.io/projected/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-kube-api-access-xkkrn\") pod \"e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.544687 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.544580 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:17.625187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.625158 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:17.795481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:17.795397 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:18.049413 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:43:18.049382 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1f0129_11cb_4e2b_a6bc_d04f12b3b284.slice/crio-7104e5cc96c1bd0022bab3a99b210c2f5577d2e66482d1f31e2fa2f6769492e5 WatchSource:0}: Error finding container 7104e5cc96c1bd0022bab3a99b210c2f5577d2e66482d1f31e2fa2f6769492e5: Status 404 returned error can't find the container with id 7104e5cc96c1bd0022bab3a99b210c2f5577d2e66482d1f31e2fa2f6769492e5 Nov 02 20:43:18.052611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:18.052575 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:43:18.052723 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:18.052649 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:43:18.052723 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:18.052678 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:43:18.072803 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:18.072780 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr"] Nov 02 20:43:18.584705 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:18.584669 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" event={"ID":"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284","Type":"ContainerStarted","Data":"5cd41de231662603c3d546f5a38b51de3af269c7b4d46c17c254ac92116ddf07"} Nov 02 20:43:18.584905 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:18.584719 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" event={"ID":"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284","Type":"ContainerStarted","Data":"7104e5cc96c1bd0022bab3a99b210c2f5577d2e66482d1f31e2fa2f6769492e5"} Nov 02 20:43:18.591100 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:18.591054 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:18.625310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:18.625289 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:18.793282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:18.793232 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" podStartSLOduration=2.793216303 podStartE2EDuration="2.793216303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:43:18.744632161 +0000 UTC m=+1727.034288688" watchObservedRunningTime="2026-11-02 20:43:18.793216303 +0000 UTC m=+1727.082872831" Nov 02 20:43:18.837022 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:18.836948 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:19.271771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:19.271737 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:19.518212 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:19.518162 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:19.631618 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:19.631595 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:20.601741 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:20.601712 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:20.602668 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:20.602651 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:43:21.645343 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:21.645310 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:21.894383 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:21.894350 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:22.048534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:22.048457 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk"] Nov 02 20:43:22.095992 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:22.095957 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf374d6f-mfcxk"] Nov 02 20:43:22.279857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:22.279828 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7"] Nov 02 20:43:22.663888 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:22.663854 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:22.915736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:22.915643 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:23.893524 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:23.890787 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d647dca8-f937-4a30-a7c6-c7039ffbf9b4" path="/var/lib/kubelet/pods/d647dca8-f937-4a30-a7c6-c7039ffbf9b4/volumes" Nov 02 20:43:24.970389 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:24.970294 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:25.688004 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:25.687962 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:25.947213 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:25.947140 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:26.534133 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:26.528783 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:26.801635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:26.801470 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:26.886448 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:26.886398 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc"] Nov 02 20:43:26.892474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:26.892403 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc" Nov 02 20:43:26.945266 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:26.945236 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc"] Nov 02 20:43:26.945695 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:26.945673 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:43:26.985433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:26.985387 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:43:26.995422 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:26.995384 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjbz8\" (UniqueName: \"kubernetes.io/projected/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-kube-api-access-rjbz8\") pod \"maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc\" (UID: \"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc" Nov 02 20:43:26.995606 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:26.995568 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc\" (UID: \"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc" Nov 02 20:43:26.995682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:26.995634 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc\" (UID: \"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc" Nov 02 20:43:27.111268 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:27.110343 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjbz8\" (UniqueName: \"kubernetes.io/projected/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-kube-api-access-rjbz8\") pod \"maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc\" (UID: \"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc" Nov 02 20:43:27.111268 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:27.110502 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc\" (UID: \"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc" Nov 02 20:43:27.111268 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:27.110556 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc\" (UID: \"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc" Nov 02 20:43:27.111580 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:27.111447 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc\" (UID: \"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc" Nov 02 20:43:27.179103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:27.179012 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjbz8\" (UniqueName: \"kubernetes.io/projected/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-kube-api-access-rjbz8\") pod \"maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc\" (UID: \"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc" Nov 02 20:43:27.232149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:27.199860 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc\" (UID: \"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc" Nov 02 20:43:27.232149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:27.210347 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc" Nov 02 20:43:27.280808 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:27.280767 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" podUID="7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e" containerName="payload-pre-processing" containerID="cri-o://36ad480acfe3810928acd5502e2c3d2a0506f7451a2610ee5fb02856923ecf43" gracePeriod=25 Nov 02 20:43:27.610728 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:43:27.610657 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1af677d_851c_4d0a_a4ae_ee7fa6c3f974.slice/crio-357285d897e460852a00bba9a0e2b7a5bec77fd6b5f6f056fb15204d62dc8331 WatchSource:0}: Error finding container 357285d897e460852a00bba9a0e2b7a5bec77fd6b5f6f056fb15204d62dc8331: Status 404 returned error can't find the container with id 357285d897e460852a00bba9a0e2b7a5bec77fd6b5f6f056fb15204d62dc8331 Nov 02 20:43:27.611143 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:27.610870 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc"] Nov 02 20:43:27.721918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:27.721886 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc" event={"ID":"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974","Type":"ContainerStarted","Data":"357285d897e460852a00bba9a0e2b7a5bec77fd6b5f6f056fb15204d62dc8331"} Nov 02 20:43:28.735929 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:28.735888 4971 generic.go:356] "Generic (PLEG): container finished" podID="e1af677d-851c-4d0a-a4ae-ee7fa6c3f974" containerID="a5ceee3a2ba723d349a076a1657f527104553a487956fb34ee30b6a4ee94e18c" exitCode=0 Nov 02 20:43:28.736474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:28.735986 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc" event={"ID":"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974","Type":"ContainerDied","Data":"a5ceee3a2ba723d349a076a1657f527104553a487956fb34ee30b6a4ee94e18c"} Nov 02 20:43:29.579635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:29.579598 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" containerID="cri-o://2695c56b339bd07f3129eb09bdda913e811d1103eb54b08ffbdf192768e58a0e" gracePeriod=45 Nov 02 20:43:29.624030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:29.624000 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" containerID="cri-o://6c17b3380b325eb376b5178e3ad199262306235c04697777d3f91370e7194bb9" gracePeriod=45 Nov 02 20:43:29.748187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:29.748139 4971 generic.go:356] "Generic (PLEG): container finished" podID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerID="2695c56b339bd07f3129eb09bdda913e811d1103eb54b08ffbdf192768e58a0e" exitCode=0 Nov 02 20:43:29.748187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:29.748167 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" event={"ID":"5c20ca15-e559-4eb5-a92c-c0049a6e0308","Type":"ContainerDied","Data":"2695c56b339bd07f3129eb09bdda913e811d1103eb54b08ffbdf192768e58a0e"} Nov 02 20:43:29.750347 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:29.750321 4971 generic.go:356] "Generic (PLEG): container finished" podID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerID="6c17b3380b325eb376b5178e3ad199262306235c04697777d3f91370e7194bb9" exitCode=0 Nov 02 20:43:29.750465 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:29.750356 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" event={"ID":"b1b99b04-38f8-4120-a237-8fc9e60071ed","Type":"ContainerDied","Data":"6c17b3380b325eb376b5178e3ad199262306235c04697777d3f91370e7194bb9"} Nov 02 20:43:29.974334 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:29.974313 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:43:29.981225 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:29.981203 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:43:29.987986 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:29.987960 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc" Nov 02 20:43:30.056355 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.056325 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-model-cache\") pod \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " Nov 02 20:43:30.056549 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.056395 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-model-cache\") pod \"b1b99b04-38f8-4120-a237-8fc9e60071ed\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " Nov 02 20:43:30.056549 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.056456 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-tmp-dir\") pod \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " Nov 02 20:43:30.056549 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.056505 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-kube-api-access-rjbz8\" (UniqueName: \"kubernetes.io/projected/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-kube-api-access-rjbz8\") pod \"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974\" (UID: \"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974\") " Nov 02 20:43:30.056736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.056550 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c20ca15-e559-4eb5-a92c-c0049a6e0308-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c20ca15-e559-4eb5-a92c-c0049a6e0308-tls-certs\") pod \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " Nov 02 20:43:30.056736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.056586 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-dshm\") pod \"b1b99b04-38f8-4120-a237-8fc9e60071ed\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " Nov 02 20:43:30.056736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.056607 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-model-cache" pod "5c20ca15-e559-4eb5-a92c-c0049a6e0308" (UID: "5c20ca15-e559-4eb5-a92c-c0049a6e0308"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:30.056736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.056686 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-model-cache" pod "b1b99b04-38f8-4120-a237-8fc9e60071ed" (UID: "b1b99b04-38f8-4120-a237-8fc9e60071ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:30.056736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.056721 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-tmp-dir\") pod \"b1b99b04-38f8-4120-a237-8fc9e60071ed\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " Nov 02 20:43:30.057228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.056761 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-home\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-home\") pod \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " Nov 02 20:43:30.057228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.056823 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1b99b04-38f8-4120-a237-8fc9e60071ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1b99b04-38f8-4120-a237-8fc9e60071ed-tls-certs\") pod \"b1b99b04-38f8-4120-a237-8fc9e60071ed\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " Nov 02 20:43:30.057228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.056849 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-dshm\") pod \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " Nov 02 20:43:30.057228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.056914 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-tmp-dir" pod "5c20ca15-e559-4eb5-a92c-c0049a6e0308" (UID: "5c20ca15-e559-4eb5-a92c-c0049a6e0308"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:30.057228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.056919 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c20ca15-e559-4eb5-a92c-c0049a6e0308-kube-api-access-mnnpk\" (UniqueName: \"kubernetes.io/projected/5c20ca15-e559-4eb5-a92c-c0049a6e0308-kube-api-access-mnnpk\") pod \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\" (UID: \"5c20ca15-e559-4eb5-a92c-c0049a6e0308\") " Nov 02 20:43:30.057228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.056975 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-maas-api-xks-ca\") pod \"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974\" (UID: \"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974\") " Nov 02 20:43:30.057228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.057012 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b99b04-38f8-4120-a237-8fc9e60071ed-kube-api-access-8lkw8\" (UniqueName: \"kubernetes.io/projected/b1b99b04-38f8-4120-a237-8fc9e60071ed-kube-api-access-8lkw8\") pod \"b1b99b04-38f8-4120-a237-8fc9e60071ed\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " Nov 02 20:43:30.057228 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.057097 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-maas-api-openshift-ca\") pod \"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974\" (UID: \"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974\") " Nov 02 20:43:30.058054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.057969 4971 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-model-cache\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.058054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.057995 4971 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-model-cache\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.058054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.058009 4971 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-tmp-dir\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.058390 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.058367 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-maas-api-openshift-ca" pod "e1af677d-851c-4d0a-a4ae-ee7fa6c3f974" (UID: "e1af677d-851c-4d0a-a4ae-ee7fa6c3f974"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:30.059676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.059643 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-kube-api-access-rjbz8" pod "e1af677d-851c-4d0a-a4ae-ee7fa6c3f974" (UID: "e1af677d-851c-4d0a-a4ae-ee7fa6c3f974"). InnerVolumeSpecName "kube-api-access-rjbz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:30.060004 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.059976 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-tmp-dir" pod "b1b99b04-38f8-4120-a237-8fc9e60071ed" (UID: "b1b99b04-38f8-4120-a237-8fc9e60071ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:30.060322 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.060298 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-home" pod "5c20ca15-e559-4eb5-a92c-c0049a6e0308" (UID: "5c20ca15-e559-4eb5-a92c-c0049a6e0308"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:30.060419 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.060401 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c20ca15-e559-4eb5-a92c-c0049a6e0308-kube-api-access-mnnpk" pod "5c20ca15-e559-4eb5-a92c-c0049a6e0308" (UID: "5c20ca15-e559-4eb5-a92c-c0049a6e0308"). InnerVolumeSpecName "kube-api-access-mnnpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:30.065066 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.063348 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c20ca15-e559-4eb5-a92c-c0049a6e0308-tls-certs" pod "5c20ca15-e559-4eb5-a92c-c0049a6e0308" (UID: "5c20ca15-e559-4eb5-a92c-c0049a6e0308"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:30.065187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.065158 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-dshm" pod "b1b99b04-38f8-4120-a237-8fc9e60071ed" (UID: "b1b99b04-38f8-4120-a237-8fc9e60071ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:30.068542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.068514 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-maas-api-xks-ca" pod "e1af677d-851c-4d0a-a4ae-ee7fa6c3f974" (UID: "e1af677d-851c-4d0a-a4ae-ee7fa6c3f974"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:30.068542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.068528 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b99b04-38f8-4120-a237-8fc9e60071ed-tls-certs" pod "b1b99b04-38f8-4120-a237-8fc9e60071ed" (UID: "b1b99b04-38f8-4120-a237-8fc9e60071ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:30.068710 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.068559 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b99b04-38f8-4120-a237-8fc9e60071ed-kube-api-access-8lkw8" pod "b1b99b04-38f8-4120-a237-8fc9e60071ed" (UID: "b1b99b04-38f8-4120-a237-8fc9e60071ed"). InnerVolumeSpecName "kube-api-access-8lkw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:30.070398 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.070370 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-dshm" pod "5c20ca15-e559-4eb5-a92c-c0049a6e0308" (UID: "5c20ca15-e559-4eb5-a92c-c0049a6e0308"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:30.158851 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.158824 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-home\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-home\") pod \"b1b99b04-38f8-4120-a237-8fc9e60071ed\" (UID: \"b1b99b04-38f8-4120-a237-8fc9e60071ed\") " Nov 02 20:43:30.159143 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.159116 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-home" pod "b1b99b04-38f8-4120-a237-8fc9e60071ed" (UID: "b1b99b04-38f8-4120-a237-8fc9e60071ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:30.159620 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.159600 4971 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-tmp-dir\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.159670 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.159629 4971 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-home\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.159670 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.159642 4971 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1b99b04-38f8-4120-a237-8fc9e60071ed-tls-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.159670 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.159655 4971 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c20ca15-e559-4eb5-a92c-c0049a6e0308-dshm\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.159770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.159668 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnnpk\" (UniqueName: \"kubernetes.io/projected/5c20ca15-e559-4eb5-a92c-c0049a6e0308-kube-api-access-mnnpk\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.159770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.159683 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.159770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.159697 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lkw8\" (UniqueName: \"kubernetes.io/projected/b1b99b04-38f8-4120-a237-8fc9e60071ed-kube-api-access-8lkw8\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.159770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.159710 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.159770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.159723 4971 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-home\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.159770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.159734 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjbz8\" (UniqueName: \"kubernetes.io/projected/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974-kube-api-access-rjbz8\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.159770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.159747 4971 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c20ca15-e559-4eb5-a92c-c0049a6e0308-tls-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.159770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.159758 4971 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1b99b04-38f8-4120-a237-8fc9e60071ed-dshm\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.481312 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.481209 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:30.760595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.760520 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc" Nov 02 20:43:30.760595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.760564 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc" event={"ID":"e1af677d-851c-4d0a-a4ae-ee7fa6c3f974","Type":"ContainerDied","Data":"357285d897e460852a00bba9a0e2b7a5bec77fd6b5f6f056fb15204d62dc8331"} Nov 02 20:43:30.761131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.760614 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="357285d897e460852a00bba9a0e2b7a5bec77fd6b5f6f056fb15204d62dc8331" Nov 02 20:43:30.762432 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.762411 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" Nov 02 20:43:30.762583 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.762451 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7" event={"ID":"5c20ca15-e559-4eb5-a92c-c0049a6e0308","Type":"ContainerDied","Data":"b411e306a8126a3dbd841abd62af93954f191dd95e5ea53cd975a77d57bfb8c1"} Nov 02 20:43:30.762583 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.762499 4971 scope.go:122] "RemoveContainer" containerID="2695c56b339bd07f3129eb09bdda913e811d1103eb54b08ffbdf192768e58a0e" Nov 02 20:43:30.764050 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.764031 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" Nov 02 20:43:30.764231 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.764073 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7" event={"ID":"b1b99b04-38f8-4120-a237-8fc9e60071ed","Type":"ContainerDied","Data":"3f2101d8c9e37396ab66f05dcfcc0699d25953893c6d6aa29801a122604299e8"} Nov 02 20:43:30.776884 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.776867 4971 scope.go:122] "RemoveContainer" containerID="6c17b3380b325eb376b5178e3ad199262306235c04697777d3f91370e7194bb9" Nov 02 20:43:30.969922 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.969870 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7"] Nov 02 20:43:30.974556 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:30.974534 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-694cee/simulator-w2-694cee-kserve-78b55d74dc-5gcn7"] Nov 02 20:43:31.077313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:31.077241 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7"] Nov 02 20:43:31.081343 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:31.081320 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-694cee/distinct-2-w2-694cee-kserve-6b7b685cdb-86sh7"] Nov 02 20:43:31.666756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:31.666713 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:31.868628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:31.868599 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" path="/var/lib/kubelet/pods/5c20ca15-e559-4eb5-a92c-c0049a6e0308/volumes" Nov 02 20:43:31.869032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:31.869017 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" path="/var/lib/kubelet/pods/b1b99b04-38f8-4120-a237-8fc9e60071ed/volumes" Nov 02 20:43:33.049772 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:33.049737 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q"] Nov 02 20:43:33.095475 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:33.095443 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a75f6cce-v626q"] Nov 02 20:43:33.340956 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:43:33.340879 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:43:33.869075 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:33.869047 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="508e995b-a379-439b-a5a1-92e2d46214f8" path="/var/lib/kubelet/pods/508e995b-a379-439b-a5a1-92e2d46214f8/volumes" Nov 02 20:43:34.728921 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:34.728880 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" podUID="7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:43:35.558423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:35.558382 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:37.620518 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:37.620476 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:37.868184 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:37.868143 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:38.118961 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:38.118931 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:38.400776 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:38.400704 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8"] Nov 02 20:43:38.400961 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:38.400892 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" podUID="cb7f2447-9d5a-4901-865e-9f61bff9b740" containerName="istio-proxy" containerID="cri-o://33245f28b7fb1691ee59e2b16fbe23f2bee5289eb6413c5ef83e7cea410f0936" gracePeriod=30 Nov 02 20:43:38.453609 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:38.453578 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22"] Nov 02 20:43:38.453860 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:38.453828 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" podUID="413b0cf5-5c2e-4983-8437-7f9d2e899405" containerName="istio-proxy" containerID="cri-o://adfcb49e6fabffbe0ae0d2556837a3c35acc2d91de68e0c02187fdcf85162583" gracePeriod=30 Nov 02 20:43:38.459241 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:38.459218 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v"] Nov 02 20:43:38.459418 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:38.459398 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" podUID="04a6c564-cef2-4570-b7d9-744b614c39d6" containerName="istio-proxy" containerID="cri-o://e6c912fb3f87832a7e43d5d546b1cf99396a4a248d03a316e7dc0503ac8c210d" gracePeriod=30 Nov 02 20:43:38.497190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:38.497158 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s"] Nov 02 20:43:38.497392 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:38.497366 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" podUID="6f9059c2-ef01-4073-9478-66013d3bf4cc" containerName="istio-proxy" containerID="cri-o://79bb69c75945b88876012b02266fdbe8fa747d610edfd1cdf3da81b80f61c982" gracePeriod=30 Nov 02 20:43:40.863451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:40.863413 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:41.395939 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:41.395910 4971 scope.go:122] "RemoveContainer" containerID="679f71de21902e84040206a438e42c6156fd2cb60c0bf42b6edf1793891e5244" Nov 02 20:43:41.407736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:41.407713 4971 scope.go:122] "RemoveContainer" containerID="4b0364b5d023fb3d5c6450537b8fc969e343eee7fdc3b988ee65aa4c836f3c06" Nov 02 20:43:41.417740 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:41.417715 4971 scope.go:122] "RemoveContainer" containerID="3de6b3bcd773ecc5c88c7ec8d9d2269c36e9934777ac1187e69206f42b9e0841" Nov 02 20:43:41.427249 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:41.427228 4971 scope.go:122] "RemoveContainer" containerID="45f2d198b2342692e20c55e21228c9725065f98ffc92341f151574c6dc465122" Nov 02 20:43:41.437160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:41.437140 4971 scope.go:122] "RemoveContainer" containerID="f71d2e66e5a819055c640a388c71f20cb49fb642a2e004df2f7e5c87a4a7c249" Nov 02 20:43:41.446901 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:41.446885 4971 scope.go:122] "RemoveContainer" containerID="9bb50f82b71240c914fef16ba282343044bc995a659e444431f57404240117f0" Nov 02 20:43:41.456283 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:41.456264 4971 scope.go:122] "RemoveContainer" containerID="cf161819604cf28ffff9a606c918eeff13b147a97a06eb58286f0e968b2a4a14" Nov 02 20:43:41.465551 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:41.465523 4971 scope.go:122] "RemoveContainer" containerID="e160a981af997f0e1c0cebd77cd8910ac47a175beb34796268ed0714aeaab43b" Nov 02 20:43:41.474901 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:41.474878 4971 scope.go:122] "RemoveContainer" containerID="922776ecef14e0cdd0fde89871398cc3f6e30abf171506e33efaf6790c64c6ef" Nov 02 20:43:41.484178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:41.484157 4971 scope.go:122] "RemoveContainer" containerID="59a13af0e62d783234ed9ffc6e963a4659bfd4c851e6f8a574fc11795bd9db43" Nov 02 20:43:41.493787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:41.493765 4971 scope.go:122] "RemoveContainer" containerID="c6e9487e761ad1c9d0ed5254307cb496e8c75c5e56ca4e32ccfc06e8e35b38b2" Nov 02 20:43:41.503063 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:41.503042 4971 scope.go:122] "RemoveContainer" containerID="a62329ac90b782238c5d7b240ae8b1e143a51e1801ce393f21cdcab15d0be172" Nov 02 20:43:41.512368 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:41.512352 4971 scope.go:122] "RemoveContainer" containerID="89fa4906eaaf3e2a42089aab00b56fb8b8072d74a0357eb47e25b2f34379f5dd" Nov 02 20:43:41.524526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:41.524509 4971 scope.go:122] "RemoveContainer" containerID="421bc6d758629abcc0fc3488118ce570127911f2efb44020a12bbd7cf46adbf1" Nov 02 20:43:41.533550 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:41.533533 4971 scope.go:122] "RemoveContainer" containerID="f498f98d1d843f8c36327cafff6fab0d705b50d8da57c668b54b17c381849fcd" Nov 02 20:43:41.542746 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:41.542731 4971 scope.go:122] "RemoveContainer" containerID="04c831b82de2675d821b6146fa69b57e3cfca0ab14563e03f6129e623d555859" Nov 02 20:43:43.810456 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.809721 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:43:43.837511 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.837484 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:43:43.893475 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.893443 4971 generic.go:356] "Generic (PLEG): container finished" podID="cb7f2447-9d5a-4901-865e-9f61bff9b740" containerID="33245f28b7fb1691ee59e2b16fbe23f2bee5289eb6413c5ef83e7cea410f0936" exitCode=0 Nov 02 20:43:43.893642 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.893484 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" Nov 02 20:43:43.893642 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.893493 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:43:43.893642 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.893523 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" event={"ID":"cb7f2447-9d5a-4901-865e-9f61bff9b740","Type":"ContainerDied","Data":"33245f28b7fb1691ee59e2b16fbe23f2bee5289eb6413c5ef83e7cea410f0936"} Nov 02 20:43:43.893642 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.893583 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8" event={"ID":"cb7f2447-9d5a-4901-865e-9f61bff9b740","Type":"ContainerDied","Data":"beb1cb40ed1d9373832aa0304235f790c87c7c8720db3a6be5947d6db53b7993"} Nov 02 20:43:43.893642 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.893605 4971 scope.go:122] "RemoveContainer" containerID="33245f28b7fb1691ee59e2b16fbe23f2bee5289eb6413c5ef83e7cea410f0936" Nov 02 20:43:43.895298 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.895280 4971 generic.go:356] "Generic (PLEG): container finished" podID="04a6c564-cef2-4570-b7d9-744b614c39d6" containerID="e6c912fb3f87832a7e43d5d546b1cf99396a4a248d03a316e7dc0503ac8c210d" exitCode=0 Nov 02 20:43:43.895397 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.895351 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" event={"ID":"04a6c564-cef2-4570-b7d9-744b614c39d6","Type":"ContainerDied","Data":"e6c912fb3f87832a7e43d5d546b1cf99396a4a248d03a316e7dc0503ac8c210d"} Nov 02 20:43:43.895397 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.895392 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" event={"ID":"04a6c564-cef2-4570-b7d9-744b614c39d6","Type":"ContainerDied","Data":"4ae737f4beef8f14288b39c9a45c2feac34c085a8115c477036840a306d196cd"} Nov 02 20:43:43.897335 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.897317 4971 generic.go:356] "Generic (PLEG): container finished" podID="6f9059c2-ef01-4073-9478-66013d3bf4cc" containerID="79bb69c75945b88876012b02266fdbe8fa747d610edfd1cdf3da81b80f61c982" exitCode=0 Nov 02 20:43:43.897436 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.897402 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" event={"ID":"6f9059c2-ef01-4073-9478-66013d3bf4cc","Type":"ContainerDied","Data":"79bb69c75945b88876012b02266fdbe8fa747d610edfd1cdf3da81b80f61c982"} Nov 02 20:43:43.899956 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.899928 4971 generic.go:356] "Generic (PLEG): container finished" podID="413b0cf5-5c2e-4983-8437-7f9d2e899405" containerID="adfcb49e6fabffbe0ae0d2556837a3c35acc2d91de68e0c02187fdcf85162583" exitCode=0 Nov 02 20:43:43.900063 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.899978 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" event={"ID":"413b0cf5-5c2e-4983-8437-7f9d2e899405","Type":"ContainerDied","Data":"adfcb49e6fabffbe0ae0d2556837a3c35acc2d91de68e0c02187fdcf85162583"} Nov 02 20:43:43.900063 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.900013 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" event={"ID":"413b0cf5-5c2e-4983-8437-7f9d2e899405","Type":"ContainerDied","Data":"b5a5dbbd3f8974d95215d74f344be960d4bbbbc4ee86cbf71113c965e8b4ab03"} Nov 02 20:43:43.900203 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.900109 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22" Nov 02 20:43:43.909068 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.909047 4971 scope.go:122] "RemoveContainer" containerID="33245f28b7fb1691ee59e2b16fbe23f2bee5289eb6413c5ef83e7cea410f0936" Nov 02 20:43:43.909385 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:43:43.909363 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33245f28b7fb1691ee59e2b16fbe23f2bee5289eb6413c5ef83e7cea410f0936\": container with ID starting with 33245f28b7fb1691ee59e2b16fbe23f2bee5289eb6413c5ef83e7cea410f0936 not found: ID does not exist" containerID="33245f28b7fb1691ee59e2b16fbe23f2bee5289eb6413c5ef83e7cea410f0936" Nov 02 20:43:43.909458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.909397 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33245f28b7fb1691ee59e2b16fbe23f2bee5289eb6413c5ef83e7cea410f0936"} err="failed to get container status \"33245f28b7fb1691ee59e2b16fbe23f2bee5289eb6413c5ef83e7cea410f0936\": rpc error: code = NotFound desc = could not find container \"33245f28b7fb1691ee59e2b16fbe23f2bee5289eb6413c5ef83e7cea410f0936\": container with ID starting with 33245f28b7fb1691ee59e2b16fbe23f2bee5289eb6413c5ef83e7cea410f0936 not found: ID does not exist" Nov 02 20:43:43.909458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.909412 4971 scope.go:122] "RemoveContainer" containerID="e6c912fb3f87832a7e43d5d546b1cf99396a4a248d03a316e7dc0503ac8c210d" Nov 02 20:43:43.916682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.916663 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb7f2447-9d5a-4901-865e-9f61bff9b740-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb7f2447-9d5a-4901-865e-9f61bff9b740-istiod-ca-cert\") pod \"cb7f2447-9d5a-4901-865e-9f61bff9b740\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " Nov 02 20:43:43.916787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.916732 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-podinfo\") pod \"cb7f2447-9d5a-4901-865e-9f61bff9b740\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " Nov 02 20:43:43.916857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.916838 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-data\") pod \"cb7f2447-9d5a-4901-865e-9f61bff9b740\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " Nov 02 20:43:43.916916 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.916887 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-workload-certs\") pod \"cb7f2447-9d5a-4901-865e-9f61bff9b740\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " Nov 02 20:43:43.916977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.916950 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-credential-socket\") pod \"cb7f2447-9d5a-4901-865e-9f61bff9b740\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " Nov 02 20:43:43.917364 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.916988 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-token\" (UniqueName: \"kubernetes.io/projected/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-token\") pod \"cb7f2447-9d5a-4901-865e-9f61bff9b740\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " Nov 02 20:43:43.917467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.917414 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb7f2447-9d5a-4901-865e-9f61bff9b740-kube-api-access-g42d8\" (UniqueName: \"kubernetes.io/projected/cb7f2447-9d5a-4901-865e-9f61bff9b740-kube-api-access-g42d8\") pod \"cb7f2447-9d5a-4901-865e-9f61bff9b740\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " Nov 02 20:43:43.917467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.917175 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-data" pod "cb7f2447-9d5a-4901-865e-9f61bff9b740" (UID: "cb7f2447-9d5a-4901-865e-9f61bff9b740"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:43.917467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.917052 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb7f2447-9d5a-4901-865e-9f61bff9b740-istiod-ca-cert" pod "cb7f2447-9d5a-4901-865e-9f61bff9b740" (UID: "cb7f2447-9d5a-4901-865e-9f61bff9b740"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:43.917467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.917255 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-credential-socket" pod "cb7f2447-9d5a-4901-865e-9f61bff9b740" (UID: "cb7f2447-9d5a-4901-865e-9f61bff9b740"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:43.917467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.917271 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-workload-certs" pod "cb7f2447-9d5a-4901-865e-9f61bff9b740" (UID: "cb7f2447-9d5a-4901-865e-9f61bff9b740"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:43.917786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.917595 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-envoy\") pod \"cb7f2447-9d5a-4901-865e-9f61bff9b740\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " Nov 02 20:43:43.917786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.917630 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-workload-socket\") pod \"cb7f2447-9d5a-4901-865e-9f61bff9b740\" (UID: \"cb7f2447-9d5a-4901-865e-9f61bff9b740\") " Nov 02 20:43:43.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.917927 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-workload-socket" pod "cb7f2447-9d5a-4901-865e-9f61bff9b740" (UID: "cb7f2447-9d5a-4901-865e-9f61bff9b740"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:43.918637 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.918611 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.918737 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.918643 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb7f2447-9d5a-4901-865e-9f61bff9b740-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.918737 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.918659 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.918737 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.918687 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.918737 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.918702 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.918938 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.918852 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:43:43.919481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.919450 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-podinfo" pod "cb7f2447-9d5a-4901-865e-9f61bff9b740" (UID: "cb7f2447-9d5a-4901-865e-9f61bff9b740"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:43.919843 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.919817 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-token" pod "cb7f2447-9d5a-4901-865e-9f61bff9b740" (UID: "cb7f2447-9d5a-4901-865e-9f61bff9b740"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:43.920009 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.919987 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-envoy" pod "cb7f2447-9d5a-4901-865e-9f61bff9b740" (UID: "cb7f2447-9d5a-4901-865e-9f61bff9b740"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:43.920234 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.920215 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7f2447-9d5a-4901-865e-9f61bff9b740-kube-api-access-g42d8" pod "cb7f2447-9d5a-4901-865e-9f61bff9b740" (UID: "cb7f2447-9d5a-4901-865e-9f61bff9b740"). InnerVolumeSpecName "kube-api-access-g42d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:43.920980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.920956 4971 scope.go:122] "RemoveContainer" containerID="e6c912fb3f87832a7e43d5d546b1cf99396a4a248d03a316e7dc0503ac8c210d" Nov 02 20:43:43.921509 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:43:43.921486 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6c912fb3f87832a7e43d5d546b1cf99396a4a248d03a316e7dc0503ac8c210d\": container with ID starting with e6c912fb3f87832a7e43d5d546b1cf99396a4a248d03a316e7dc0503ac8c210d not found: ID does not exist" containerID="e6c912fb3f87832a7e43d5d546b1cf99396a4a248d03a316e7dc0503ac8c210d" Nov 02 20:43:43.921576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.921521 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c912fb3f87832a7e43d5d546b1cf99396a4a248d03a316e7dc0503ac8c210d"} err="failed to get container status \"e6c912fb3f87832a7e43d5d546b1cf99396a4a248d03a316e7dc0503ac8c210d\": rpc error: code = NotFound desc = could not find container \"e6c912fb3f87832a7e43d5d546b1cf99396a4a248d03a316e7dc0503ac8c210d\": container with ID starting with e6c912fb3f87832a7e43d5d546b1cf99396a4a248d03a316e7dc0503ac8c210d not found: ID does not exist" Nov 02 20:43:43.921576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.921538 4971 scope.go:122] "RemoveContainer" containerID="adfcb49e6fabffbe0ae0d2556837a3c35acc2d91de68e0c02187fdcf85162583" Nov 02 20:43:43.932311 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.932296 4971 scope.go:122] "RemoveContainer" containerID="327b455ca520a38bbc20f7043c200b3fc50952ad9e43941dacd42e699668857e" Nov 02 20:43:43.943154 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.943137 4971 scope.go:122] "RemoveContainer" containerID="adfcb49e6fabffbe0ae0d2556837a3c35acc2d91de68e0c02187fdcf85162583" Nov 02 20:43:43.943399 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:43:43.943373 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adfcb49e6fabffbe0ae0d2556837a3c35acc2d91de68e0c02187fdcf85162583\": container with ID starting with adfcb49e6fabffbe0ae0d2556837a3c35acc2d91de68e0c02187fdcf85162583 not found: ID does not exist" containerID="adfcb49e6fabffbe0ae0d2556837a3c35acc2d91de68e0c02187fdcf85162583" Nov 02 20:43:43.943476 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.943404 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adfcb49e6fabffbe0ae0d2556837a3c35acc2d91de68e0c02187fdcf85162583"} err="failed to get container status \"adfcb49e6fabffbe0ae0d2556837a3c35acc2d91de68e0c02187fdcf85162583\": rpc error: code = NotFound desc = could not find container \"adfcb49e6fabffbe0ae0d2556837a3c35acc2d91de68e0c02187fdcf85162583\": container with ID starting with adfcb49e6fabffbe0ae0d2556837a3c35acc2d91de68e0c02187fdcf85162583 not found: ID does not exist" Nov 02 20:43:43.943476 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.943425 4971 scope.go:122] "RemoveContainer" containerID="327b455ca520a38bbc20f7043c200b3fc50952ad9e43941dacd42e699668857e" Nov 02 20:43:43.943642 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:43:43.943615 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"327b455ca520a38bbc20f7043c200b3fc50952ad9e43941dacd42e699668857e\": container with ID starting with 327b455ca520a38bbc20f7043c200b3fc50952ad9e43941dacd42e699668857e not found: ID does not exist" containerID="327b455ca520a38bbc20f7043c200b3fc50952ad9e43941dacd42e699668857e" Nov 02 20:43:43.943703 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:43.943642 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"327b455ca520a38bbc20f7043c200b3fc50952ad9e43941dacd42e699668857e"} err="failed to get container status \"327b455ca520a38bbc20f7043c200b3fc50952ad9e43941dacd42e699668857e\": rpc error: code = NotFound desc = could not find container \"327b455ca520a38bbc20f7043c200b3fc50952ad9e43941dacd42e699668857e\": container with ID starting with 327b455ca520a38bbc20f7043c200b3fc50952ad9e43941dacd42e699668857e not found: ID does not exist" Nov 02 20:43:44.019453 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.019370 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-workload-socket\") pod \"6f9059c2-ef01-4073-9478-66013d3bf4cc\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " Nov 02 20:43:44.019453 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.019426 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/413b0cf5-5c2e-4983-8437-7f9d2e899405-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413b0cf5-5c2e-4983-8437-7f9d2e899405-istiod-ca-cert\") pod \"413b0cf5-5c2e-4983-8437-7f9d2e899405\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " Nov 02 20:43:44.019663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.019468 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-workload-socket\") pod \"04a6c564-cef2-4570-b7d9-744b614c39d6\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " Nov 02 20:43:44.019663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.019529 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-envoy\") pod \"6f9059c2-ef01-4073-9478-66013d3bf4cc\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " Nov 02 20:43:44.019663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.019621 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-workload-socket" pod "6f9059c2-ef01-4073-9478-66013d3bf4cc" (UID: "6f9059c2-ef01-4073-9478-66013d3bf4cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.019832 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.019631 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-podinfo\") pod \"04a6c564-cef2-4570-b7d9-744b614c39d6\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " Nov 02 20:43:44.019832 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.019716 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f9059c2-ef01-4073-9478-66013d3bf4cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9059c2-ef01-4073-9478-66013d3bf4cc-istiod-ca-cert\") pod \"6f9059c2-ef01-4073-9478-66013d3bf4cc\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " Nov 02 20:43:44.019832 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.019812 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/413b0cf5-5c2e-4983-8437-7f9d2e899405-kube-api-access-62qs7\" (UniqueName: \"kubernetes.io/projected/413b0cf5-5c2e-4983-8437-7f9d2e899405-kube-api-access-62qs7\") pod \"413b0cf5-5c2e-4983-8437-7f9d2e899405\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " Nov 02 20:43:44.019832 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.019812 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413b0cf5-5c2e-4983-8437-7f9d2e899405-istiod-ca-cert" pod "413b0cf5-5c2e-4983-8437-7f9d2e899405" (UID: "413b0cf5-5c2e-4983-8437-7f9d2e899405"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:44.020039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.019854 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-workload-socket" pod "04a6c564-cef2-4570-b7d9-744b614c39d6" (UID: "04a6c564-cef2-4570-b7d9-744b614c39d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.020039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.019861 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-data\") pod \"413b0cf5-5c2e-4983-8437-7f9d2e899405\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " Nov 02 20:43:44.020039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.019920 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a6c564-cef2-4570-b7d9-744b614c39d6-kube-api-access-7jnmd\" (UniqueName: \"kubernetes.io/projected/04a6c564-cef2-4570-b7d9-744b614c39d6-kube-api-access-7jnmd\") pod \"04a6c564-cef2-4570-b7d9-744b614c39d6\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " Nov 02 20:43:44.020039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.019961 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a6c564-cef2-4570-b7d9-744b614c39d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a6c564-cef2-4570-b7d9-744b614c39d6-istiod-ca-cert\") pod \"04a6c564-cef2-4570-b7d9-744b614c39d6\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " Nov 02 20:43:44.020039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020020 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-credential-socket\") pod \"6f9059c2-ef01-4073-9478-66013d3bf4cc\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " Nov 02 20:43:44.020585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020049 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-data" pod "413b0cf5-5c2e-4983-8437-7f9d2e899405" (UID: "413b0cf5-5c2e-4983-8437-7f9d2e899405"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.020585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020075 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-envoy\") pod \"04a6c564-cef2-4570-b7d9-744b614c39d6\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " Nov 02 20:43:44.020585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020129 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-workload-certs\") pod \"413b0cf5-5c2e-4983-8437-7f9d2e899405\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " Nov 02 20:43:44.020585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020166 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-credential-socket\") pod \"04a6c564-cef2-4570-b7d9-744b614c39d6\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " Nov 02 20:43:44.020585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020206 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-envoy\") pod \"413b0cf5-5c2e-4983-8437-7f9d2e899405\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " Nov 02 20:43:44.020585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020262 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-credential-socket\") pod \"413b0cf5-5c2e-4983-8437-7f9d2e899405\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " Nov 02 20:43:44.020585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020302 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-workload-certs\") pod \"6f9059c2-ef01-4073-9478-66013d3bf4cc\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " Nov 02 20:43:44.020585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020350 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-data\") pod \"04a6c564-cef2-4570-b7d9-744b614c39d6\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " Nov 02 20:43:44.020585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020365 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9059c2-ef01-4073-9478-66013d3bf4cc-istiod-ca-cert" pod "6f9059c2-ef01-4073-9478-66013d3bf4cc" (UID: "6f9059c2-ef01-4073-9478-66013d3bf4cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:44.020585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020380 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-workload-socket\") pod \"413b0cf5-5c2e-4983-8437-7f9d2e899405\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " Nov 02 20:43:44.020585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020409 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a6c564-cef2-4570-b7d9-744b614c39d6-istiod-ca-cert" pod "04a6c564-cef2-4570-b7d9-744b614c39d6" (UID: "04a6c564-cef2-4570-b7d9-744b614c39d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:44.020585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020473 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-podinfo\") pod \"6f9059c2-ef01-4073-9478-66013d3bf4cc\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " Nov 02 20:43:44.020585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020541 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-podinfo\") pod \"413b0cf5-5c2e-4983-8437-7f9d2e899405\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " Nov 02 20:43:44.020585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020592 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-token\" (UniqueName: \"kubernetes.io/projected/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-token\") pod \"04a6c564-cef2-4570-b7d9-744b614c39d6\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020659 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-credential-socket" pod "04a6c564-cef2-4570-b7d9-744b614c39d6" (UID: "04a6c564-cef2-4570-b7d9-744b614c39d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020672 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-credential-socket" pod "413b0cf5-5c2e-4983-8437-7f9d2e899405" (UID: "413b0cf5-5c2e-4983-8437-7f9d2e899405"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020827 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-token\" (UniqueName: \"kubernetes.io/projected/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-token\") pod \"413b0cf5-5c2e-4983-8437-7f9d2e899405\" (UID: \"413b0cf5-5c2e-4983-8437-7f9d2e899405\") " Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020890 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9059c2-ef01-4073-9478-66013d3bf4cc-kube-api-access-7j8ht\" (UniqueName: \"kubernetes.io/projected/6f9059c2-ef01-4073-9478-66013d3bf4cc-kube-api-access-7j8ht\") pod \"6f9059c2-ef01-4073-9478-66013d3bf4cc\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020933 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-data\") pod \"6f9059c2-ef01-4073-9478-66013d3bf4cc\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.020975 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-workload-certs\") pod \"04a6c564-cef2-4570-b7d9-744b614c39d6\" (UID: \"04a6c564-cef2-4570-b7d9-744b614c39d6\") " Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.021007 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-token\" (UniqueName: \"kubernetes.io/projected/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-token\") pod \"6f9059c2-ef01-4073-9478-66013d3bf4cc\" (UID: \"6f9059c2-ef01-4073-9478-66013d3bf4cc\") " Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.021111 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-workload-socket" pod "413b0cf5-5c2e-4983-8437-7f9d2e899405" (UID: "413b0cf5-5c2e-4983-8437-7f9d2e899405"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.021311 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-workload-certs" pod "6f9059c2-ef01-4073-9478-66013d3bf4cc" (UID: "6f9059c2-ef01-4073-9478-66013d3bf4cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.021488 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-data" pod "04a6c564-cef2-4570-b7d9-744b614c39d6" (UID: "04a6c564-cef2-4570-b7d9-744b614c39d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.021554 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-credential-socket" pod "6f9059c2-ef01-4073-9478-66013d3bf4cc" (UID: "6f9059c2-ef01-4073-9478-66013d3bf4cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.021559 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-workload-certs" pod "413b0cf5-5c2e-4983-8437-7f9d2e899405" (UID: "413b0cf5-5c2e-4983-8437-7f9d2e899405"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.021745 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-data" pod "6f9059c2-ef01-4073-9478-66013d3bf4cc" (UID: "6f9059c2-ef01-4073-9478-66013d3bf4cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.021813 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-workload-certs" pod "04a6c564-cef2-4570-b7d9-744b614c39d6" (UID: "04a6c564-cef2-4570-b7d9-744b614c39d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022119 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022139 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413b0cf5-5c2e-4983-8437-7f9d2e899405-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022153 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022166 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9059c2-ef01-4073-9478-66013d3bf4cc-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022185 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022198 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a6c564-cef2-4570-b7d9-744b614c39d6-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022210 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022223 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022236 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022251 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g42d8\" (UniqueName: \"kubernetes.io/projected/cb7f2447-9d5a-4901-865e-9f61bff9b740-kube-api-access-g42d8\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022266 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022280 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022293 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022306 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022320 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022333 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022347 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb7f2447-9d5a-4901-865e-9f61bff9b740-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022362 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.022717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.022375 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.024730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.023418 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-podinfo" pod "04a6c564-cef2-4570-b7d9-744b614c39d6" (UID: "04a6c564-cef2-4570-b7d9-744b614c39d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:44.024730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.024005 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-envoy" pod "413b0cf5-5c2e-4983-8437-7f9d2e899405" (UID: "413b0cf5-5c2e-4983-8437-7f9d2e899405"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.024730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.024393 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-token" pod "413b0cf5-5c2e-4983-8437-7f9d2e899405" (UID: "413b0cf5-5c2e-4983-8437-7f9d2e899405"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:44.025403 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.025310 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-token" pod "6f9059c2-ef01-4073-9478-66013d3bf4cc" (UID: "6f9059c2-ef01-4073-9478-66013d3bf4cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:44.025753 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.025705 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413b0cf5-5c2e-4983-8437-7f9d2e899405-kube-api-access-62qs7" pod "413b0cf5-5c2e-4983-8437-7f9d2e899405" (UID: "413b0cf5-5c2e-4983-8437-7f9d2e899405"). InnerVolumeSpecName "kube-api-access-62qs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:44.026787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.026720 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-podinfo" pod "413b0cf5-5c2e-4983-8437-7f9d2e899405" (UID: "413b0cf5-5c2e-4983-8437-7f9d2e899405"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:44.026787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.026722 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-podinfo" pod "6f9059c2-ef01-4073-9478-66013d3bf4cc" (UID: "6f9059c2-ef01-4073-9478-66013d3bf4cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:44.026787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.026738 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-envoy" pod "04a6c564-cef2-4570-b7d9-744b614c39d6" (UID: "04a6c564-cef2-4570-b7d9-744b614c39d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.026787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.026740 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-envoy" pod "6f9059c2-ef01-4073-9478-66013d3bf4cc" (UID: "6f9059c2-ef01-4073-9478-66013d3bf4cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.027036 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.026792 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a6c564-cef2-4570-b7d9-744b614c39d6-kube-api-access-7jnmd" pod "04a6c564-cef2-4570-b7d9-744b614c39d6" (UID: "04a6c564-cef2-4570-b7d9-744b614c39d6"). InnerVolumeSpecName "kube-api-access-7jnmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:44.027036 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.026828 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9059c2-ef01-4073-9478-66013d3bf4cc-kube-api-access-7j8ht" pod "6f9059c2-ef01-4073-9478-66013d3bf4cc" (UID: "6f9059c2-ef01-4073-9478-66013d3bf4cc"). InnerVolumeSpecName "kube-api-access-7j8ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:44.027170 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.027037 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-token" pod "04a6c564-cef2-4570-b7d9-744b614c39d6" (UID: "04a6c564-cef2-4570-b7d9-744b614c39d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:44.123320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.123293 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.123320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.123318 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.123478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.123329 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.123478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.123338 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.123478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.123346 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.123478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.123355 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413b0cf5-5c2e-4983-8437-7f9d2e899405-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.123478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.123365 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j8ht\" (UniqueName: \"kubernetes.io/projected/6f9059c2-ef01-4073-9478-66013d3bf4cc-kube-api-access-7j8ht\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.123478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.123375 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.123478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.123383 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9059c2-ef01-4073-9478-66013d3bf4cc-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.123478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.123391 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a6c564-cef2-4570-b7d9-744b614c39d6-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.123478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.123399 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62qs7\" (UniqueName: \"kubernetes.io/projected/413b0cf5-5c2e-4983-8437-7f9d2e899405-kube-api-access-62qs7\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.123478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.123408 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jnmd\" (UniqueName: \"kubernetes.io/projected/04a6c564-cef2-4570-b7d9-744b614c39d6-kube-api-access-7jnmd\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.482241 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.482214 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22"] Nov 02 20:43:44.486668 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.486644 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-cdw22"] Nov 02 20:43:44.588482 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.588449 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8"] Nov 02 20:43:44.593794 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.593676 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-hcnx8"] Nov 02 20:43:44.723003 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.722927 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" podUID="7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:43:44.909441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.909402 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v" Nov 02 20:43:44.911226 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.911183 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" Nov 02 20:43:44.911226 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.911204 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s" event={"ID":"6f9059c2-ef01-4073-9478-66013d3bf4cc","Type":"ContainerDied","Data":"59398373a90daff27a93a50ef612a89fb223cd394514b06af92661d04b769244"} Nov 02 20:43:44.911428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:44.911271 4971 scope.go:122] "RemoveContainer" containerID="79bb69c75945b88876012b02266fdbe8fa747d610edfd1cdf3da81b80f61c982" Nov 02 20:43:45.073869 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.073784 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s"] Nov 02 20:43:45.078039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.078014 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-xfb8s"] Nov 02 20:43:45.180159 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.180134 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v"] Nov 02 20:43:45.185985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.185959 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-c9f5v"] Nov 02 20:43:45.459038 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.459007 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh"] Nov 02 20:43:45.459748 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.459730 4971 cpu_manager.go:414] "containerMap: removing container" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" Nov 02 20:43:45.459845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.459750 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" Nov 02 20:43:45.459845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.459767 4971 cpu_manager.go:414] "containerMap: removing container" podUID="413b0cf5-5c2e-4983-8437-7f9d2e899405" containerName="istio-proxy" Nov 02 20:43:45.459845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.459772 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="413b0cf5-5c2e-4983-8437-7f9d2e899405" containerName="istio-proxy" Nov 02 20:43:45.459845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.459796 4971 cpu_manager.go:414] "containerMap: removing container" podUID="04a6c564-cef2-4570-b7d9-744b614c39d6" containerName="istio-proxy" Nov 02 20:43:45.459845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.459801 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a6c564-cef2-4570-b7d9-744b614c39d6" containerName="istio-proxy" Nov 02 20:43:45.459845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.459829 4971 cpu_manager.go:414] "containerMap: removing container" podUID="cb7f2447-9d5a-4901-865e-9f61bff9b740" containerName="istio-proxy" Nov 02 20:43:45.459845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.459834 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb7f2447-9d5a-4901-865e-9f61bff9b740" containerName="istio-proxy" Nov 02 20:43:45.459845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.459841 4971 cpu_manager.go:414] "containerMap: removing container" podUID="e1af677d-851c-4d0a-a4ae-ee7fa6c3f974" containerName="revoke-keys" Nov 02 20:43:45.459845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.459846 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1af677d-851c-4d0a-a4ae-ee7fa6c3f974" containerName="revoke-keys" Nov 02 20:43:45.460146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.459854 4971 cpu_manager.go:414] "containerMap: removing container" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" Nov 02 20:43:45.460146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.459859 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" Nov 02 20:43:45.460146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.459866 4971 cpu_manager.go:414] "containerMap: removing container" podUID="6f9059c2-ef01-4073-9478-66013d3bf4cc" containerName="istio-proxy" Nov 02 20:43:45.460146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.459871 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9059c2-ef01-4073-9478-66013d3bf4cc" containerName="istio-proxy" Nov 02 20:43:45.460146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.460013 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="413b0cf5-5c2e-4983-8437-7f9d2e899405" containerName="istio-proxy" Nov 02 20:43:45.460146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.460037 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a6c564-cef2-4570-b7d9-744b614c39d6" containerName="istio-proxy" Nov 02 20:43:45.460146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.460060 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="413b0cf5-5c2e-4983-8437-7f9d2e899405" containerName="istio-proxy" Nov 02 20:43:45.460146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.460069 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb7f2447-9d5a-4901-865e-9f61bff9b740" containerName="istio-proxy" Nov 02 20:43:45.460146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.460076 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1af677d-851c-4d0a-a4ae-ee7fa6c3f974" containerName="revoke-keys" Nov 02 20:43:45.460146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.460098 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b99b04-38f8-4120-a237-8fc9e60071ed" containerName="main" Nov 02 20:43:45.460146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.460104 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9059c2-ef01-4073-9478-66013d3bf4cc" containerName="istio-proxy" Nov 02 20:43:45.460146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.460112 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c20ca15-e559-4eb5-a92c-c0049a6e0308" containerName="main" Nov 02 20:43:45.465479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.465461 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" Nov 02 20:43:45.513954 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.513930 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m"] Nov 02 20:43:45.514842 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.514801 4971 cpu_manager.go:414] "containerMap: removing container" podUID="413b0cf5-5c2e-4983-8437-7f9d2e899405" containerName="istio-proxy" Nov 02 20:43:45.514842 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.514825 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="413b0cf5-5c2e-4983-8437-7f9d2e899405" containerName="istio-proxy" Nov 02 20:43:45.519834 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.518367 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:43:45.519834 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.518407 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:43:45.519834 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.519321 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" Nov 02 20:43:45.520278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.520257 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh"] Nov 02 20:43:45.572110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.572068 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m"] Nov 02 20:43:45.648857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.648810 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffd55\" (UniqueName: \"kubernetes.io/projected/1c278625-8b76-4f43-95a3-510cb72ea044-kube-api-access-ffd55\") pod \"maas-api-revoke-sub-f7a4600b-7kmmh\" (UID: \"1c278625-8b76-4f43-95a3-510cb72ea044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" Nov 02 20:43:45.648857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.648862 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-893a0a2c-rhs7m\" (UID: \"3c0b5b7d-6e10-4dea-80df-69f60f219ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" Nov 02 20:43:45.649029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.648998 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c278625-8b76-4f43-95a3-510cb72ea044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7a4600b-7kmmh\" (UID: \"1c278625-8b76-4f43-95a3-510cb72ea044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" Nov 02 20:43:45.649115 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.649066 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-893a0a2c-rhs7m\" (UID: \"3c0b5b7d-6e10-4dea-80df-69f60f219ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" Nov 02 20:43:45.649187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.649169 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c278625-8b76-4f43-95a3-510cb72ea044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7a4600b-7kmmh\" (UID: \"1c278625-8b76-4f43-95a3-510cb72ea044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" Nov 02 20:43:45.649314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.649300 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh5vh\" (UniqueName: \"kubernetes.io/projected/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-kube-api-access-gh5vh\") pod \"maas-api-revoke-sub-893a0a2c-rhs7m\" (UID: \"3c0b5b7d-6e10-4dea-80df-69f60f219ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" Nov 02 20:43:45.750555 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.750479 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffd55\" (UniqueName: \"kubernetes.io/projected/1c278625-8b76-4f43-95a3-510cb72ea044-kube-api-access-ffd55\") pod \"maas-api-revoke-sub-f7a4600b-7kmmh\" (UID: \"1c278625-8b76-4f43-95a3-510cb72ea044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" Nov 02 20:43:45.750555 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.750514 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-893a0a2c-rhs7m\" (UID: \"3c0b5b7d-6e10-4dea-80df-69f60f219ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" Nov 02 20:43:45.750732 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.750574 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c278625-8b76-4f43-95a3-510cb72ea044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7a4600b-7kmmh\" (UID: \"1c278625-8b76-4f43-95a3-510cb72ea044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" Nov 02 20:43:45.750732 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.750628 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-893a0a2c-rhs7m\" (UID: \"3c0b5b7d-6e10-4dea-80df-69f60f219ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" Nov 02 20:43:45.750732 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.750676 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c278625-8b76-4f43-95a3-510cb72ea044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7a4600b-7kmmh\" (UID: \"1c278625-8b76-4f43-95a3-510cb72ea044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" Nov 02 20:43:45.750872 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.750815 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh5vh\" (UniqueName: \"kubernetes.io/projected/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-kube-api-access-gh5vh\") pod \"maas-api-revoke-sub-893a0a2c-rhs7m\" (UID: \"3c0b5b7d-6e10-4dea-80df-69f60f219ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" Nov 02 20:43:45.751379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.751355 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-893a0a2c-rhs7m\" (UID: \"3c0b5b7d-6e10-4dea-80df-69f60f219ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" Nov 02 20:43:45.751379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.751369 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c278625-8b76-4f43-95a3-510cb72ea044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7a4600b-7kmmh\" (UID: \"1c278625-8b76-4f43-95a3-510cb72ea044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" Nov 02 20:43:45.753153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.753126 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-893a0a2c-rhs7m\" (UID: \"3c0b5b7d-6e10-4dea-80df-69f60f219ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" Nov 02 20:43:45.753275 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.753242 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c278625-8b76-4f43-95a3-510cb72ea044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7a4600b-7kmmh\" (UID: \"1c278625-8b76-4f43-95a3-510cb72ea044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" Nov 02 20:43:45.808451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.808424 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffd55\" (UniqueName: \"kubernetes.io/projected/1c278625-8b76-4f43-95a3-510cb72ea044-kube-api-access-ffd55\") pod \"maas-api-revoke-sub-f7a4600b-7kmmh\" (UID: \"1c278625-8b76-4f43-95a3-510cb72ea044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" Nov 02 20:43:45.808592 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.808525 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh5vh\" (UniqueName: \"kubernetes.io/projected/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-kube-api-access-gh5vh\") pod \"maas-api-revoke-sub-893a0a2c-rhs7m\" (UID: \"3c0b5b7d-6e10-4dea-80df-69f60f219ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" Nov 02 20:43:45.835882 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.835859 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" Nov 02 20:43:45.871024 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.870999 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a6c564-cef2-4570-b7d9-744b614c39d6" path="/var/lib/kubelet/pods/04a6c564-cef2-4570-b7d9-744b614c39d6/volumes" Nov 02 20:43:45.871531 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.871515 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="413b0cf5-5c2e-4983-8437-7f9d2e899405" path="/var/lib/kubelet/pods/413b0cf5-5c2e-4983-8437-7f9d2e899405/volumes" Nov 02 20:43:45.871984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.871960 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f9059c2-ef01-4073-9478-66013d3bf4cc" path="/var/lib/kubelet/pods/6f9059c2-ef01-4073-9478-66013d3bf4cc/volumes" Nov 02 20:43:45.872772 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:45.872754 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb7f2447-9d5a-4901-865e-9f61bff9b740" path="/var/lib/kubelet/pods/cb7f2447-9d5a-4901-865e-9f61bff9b740/volumes" Nov 02 20:43:46.079907 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:46.079829 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" Nov 02 20:43:46.224861 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:43:46.224835 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0b5b7d_6e10_4dea_80df_69f60f219ff7.slice/crio-23c4f1176fd567e7d1dc6511cfd86e0d3dec56e821c8f735e84f4ca5c74b4b78 WatchSource:0}: Error finding container 23c4f1176fd567e7d1dc6511cfd86e0d3dec56e821c8f735e84f4ca5c74b4b78: Status 404 returned error can't find the container with id 23c4f1176fd567e7d1dc6511cfd86e0d3dec56e821c8f735e84f4ca5c74b4b78 Nov 02 20:43:46.226426 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:46.226402 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m"] Nov 02 20:43:46.457665 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:46.457635 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh"] Nov 02 20:43:46.458993 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:43:46.458967 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c278625_8b76_4f43_95a3_510cb72ea044.slice/crio-d27316e9a50baae4fbb643f0c93d9e81faf5b26d582f439973f48885163b7e81 WatchSource:0}: Error finding container d27316e9a50baae4fbb643f0c93d9e81faf5b26d582f439973f48885163b7e81: Status 404 returned error can't find the container with id d27316e9a50baae4fbb643f0c93d9e81faf5b26d582f439973f48885163b7e81 Nov 02 20:43:46.466121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:46.466061 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:46.937272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:46.937214 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" event={"ID":"1c278625-8b76-4f43-95a3-510cb72ea044","Type":"ContainerStarted","Data":"8c024e201f338afbab80c8ecdf0ddc8eeb9c379a3189223937a09935d5bcc8d5"} Nov 02 20:43:46.937476 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:46.937285 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" event={"ID":"1c278625-8b76-4f43-95a3-510cb72ea044","Type":"ContainerStarted","Data":"d27316e9a50baae4fbb643f0c93d9e81faf5b26d582f439973f48885163b7e81"} Nov 02 20:43:46.939693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:46.939667 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" event={"ID":"3c0b5b7d-6e10-4dea-80df-69f60f219ff7","Type":"ContainerStarted","Data":"55a21598be72020e2a01eb253514fc73387b33041d40050d35ea5d75faa2fa19"} Nov 02 20:43:46.939965 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:46.939714 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" event={"ID":"3c0b5b7d-6e10-4dea-80df-69f60f219ff7","Type":"ContainerStarted","Data":"23c4f1176fd567e7d1dc6511cfd86e0d3dec56e821c8f735e84f4ca5c74b4b78"} Nov 02 20:43:47.046595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.046476 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" podStartSLOduration=2.046459098 podStartE2EDuration="2.046459098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:43:47.046385602 +0000 UTC m=+1755.336042129" watchObservedRunningTime="2026-11-02 20:43:47.046459098 +0000 UTC m=+1755.336115626" Nov 02 20:43:47.156100 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.156029 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" podStartSLOduration=2.156016679 podStartE2EDuration="2.156016679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:43:47.154732808 +0000 UTC m=+1755.444389338" watchObservedRunningTime="2026-11-02 20:43:47.156016679 +0000 UTC m=+1755.445673208" Nov 02 20:43:47.564636 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.564612 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" Nov 02 20:43:47.574118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.574074 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e-plugins-config-volume\") pod \"7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e\" (UID: \"7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e\") " Nov 02 20:43:47.574256 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.574221 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e-kube-api-access-kc2hl\" (UniqueName: \"kubernetes.io/projected/7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e-kube-api-access-kc2hl\") pod \"7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e\" (UID: \"7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e\") " Nov 02 20:43:47.574475 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.574451 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e-plugins-config-volume" pod "7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e" (UID: "7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:47.574927 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.574906 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:47.576434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.576414 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e-kube-api-access-kc2hl" pod "7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e" (UID: "7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e"). InnerVolumeSpecName "kube-api-access-kc2hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:47.677497 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.676682 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc2hl\" (UniqueName: \"kubernetes.io/projected/7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e-kube-api-access-kc2hl\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:47.949240 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.949208 4971 generic.go:356] "Generic (PLEG): container finished" podID="3c0b5b7d-6e10-4dea-80df-69f60f219ff7" containerID="55a21598be72020e2a01eb253514fc73387b33041d40050d35ea5d75faa2fa19" exitCode=0 Nov 02 20:43:47.949400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.949301 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" event={"ID":"3c0b5b7d-6e10-4dea-80df-69f60f219ff7","Type":"ContainerDied","Data":"55a21598be72020e2a01eb253514fc73387b33041d40050d35ea5d75faa2fa19"} Nov 02 20:43:47.950677 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.950656 4971 generic.go:356] "Generic (PLEG): container finished" podID="7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e" containerID="36ad480acfe3810928acd5502e2c3d2a0506f7451a2610ee5fb02856923ecf43" exitCode=0 Nov 02 20:43:47.950780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.950694 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" Nov 02 20:43:47.950780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.950742 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" event={"ID":"7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e","Type":"ContainerDied","Data":"36ad480acfe3810928acd5502e2c3d2a0506f7451a2610ee5fb02856923ecf43"} Nov 02 20:43:47.950860 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.950791 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7" event={"ID":"7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e","Type":"ContainerDied","Data":"e672bfcd87cd1d22938d23c76f1668e65a1c93b42a2021fab5f0c756a958ea98"} Nov 02 20:43:47.950860 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.950819 4971 scope.go:122] "RemoveContainer" containerID="36ad480acfe3810928acd5502e2c3d2a0506f7451a2610ee5fb02856923ecf43" Nov 02 20:43:47.952711 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.952686 4971 generic.go:356] "Generic (PLEG): container finished" podID="1c278625-8b76-4f43-95a3-510cb72ea044" containerID="8c024e201f338afbab80c8ecdf0ddc8eeb9c379a3189223937a09935d5bcc8d5" exitCode=0 Nov 02 20:43:47.952845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.952766 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" event={"ID":"1c278625-8b76-4f43-95a3-510cb72ea044","Type":"ContainerDied","Data":"8c024e201f338afbab80c8ecdf0ddc8eeb9c379a3189223937a09935d5bcc8d5"} Nov 02 20:43:47.961767 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.961748 4971 scope.go:122] "RemoveContainer" containerID="36ad480acfe3810928acd5502e2c3d2a0506f7451a2610ee5fb02856923ecf43" Nov 02 20:43:47.962057 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:43:47.962037 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ad480acfe3810928acd5502e2c3d2a0506f7451a2610ee5fb02856923ecf43\": container with ID starting with 36ad480acfe3810928acd5502e2c3d2a0506f7451a2610ee5fb02856923ecf43 not found: ID does not exist" containerID="36ad480acfe3810928acd5502e2c3d2a0506f7451a2610ee5fb02856923ecf43" Nov 02 20:43:47.962142 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:47.962064 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ad480acfe3810928acd5502e2c3d2a0506f7451a2610ee5fb02856923ecf43"} err="failed to get container status \"36ad480acfe3810928acd5502e2c3d2a0506f7451a2610ee5fb02856923ecf43\": rpc error: code = NotFound desc = could not find container \"36ad480acfe3810928acd5502e2c3d2a0506f7451a2610ee5fb02856923ecf43\": container with ID starting with 36ad480acfe3810928acd5502e2c3d2a0506f7451a2610ee5fb02856923ecf43 not found: ID does not exist" Nov 02 20:43:48.165150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.165027 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7"] Nov 02 20:43:48.169031 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.169007 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-694cee-7bb5d66c6c-gf4z7"] Nov 02 20:43:48.478077 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.478007 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6"] Nov 02 20:43:48.478759 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.478744 4971 cpu_manager.go:414] "containerMap: removing container" podUID="7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e" containerName="payload-pre-processing" Nov 02 20:43:48.478803 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.478761 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e" containerName="payload-pre-processing" Nov 02 20:43:48.478889 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.478880 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e" containerName="payload-pre-processing" Nov 02 20:43:48.482552 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.482536 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.539649 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.539613 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-039b89-openshift-default-dockercfg-c8dmn\"" Nov 02 20:43:48.544427 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.544400 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6"] Nov 02 20:43:48.588409 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.588381 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-workload-certs\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.588535 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.588441 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-credential-socket\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.588618 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.588593 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsc5v\" (UniqueName: \"kubernetes.io/projected/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-kube-api-access-bsc5v\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.588685 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.588634 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-token\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.588876 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.588860 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-data\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.588926 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.588909 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istiod-ca-cert\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.589005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.588991 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-envoy\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.589133 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.589060 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-workload-socket\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.589199 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.589180 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-podinfo\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.672558 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.672475 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:48.690393 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.690350 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-data\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.690567 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.690417 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istiod-ca-cert\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.690567 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.690452 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-envoy\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.690567 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.690519 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-workload-socket\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.690762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.690646 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-podinfo\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.690762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.690734 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-workload-certs\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.690878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.690776 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-credential-socket\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.690878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.690833 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsc5v\" (UniqueName: \"kubernetes.io/projected/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-kube-api-access-bsc5v\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.690878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.690857 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-token\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.691045 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.690974 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-data\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.691699 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.691670 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-credential-socket\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.691914 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.691890 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-workload-socket\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.692068 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.692046 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istiod-ca-cert\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.692333 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.692303 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-workload-certs\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.693949 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.693925 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-podinfo\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.694385 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.694365 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-envoy\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.749062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.748980 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-token\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.749235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.749132 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsc5v\" (UniqueName: \"kubernetes.io/projected/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-kube-api-access-bsc5v\") pod \"e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.797525 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.797485 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:48.988450 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:48.988324 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:49.202608 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.202582 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" Nov 02 20:43:49.206517 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.206496 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" Nov 02 20:43:49.289710 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:43:49.289687 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1e4721_0a73_49fe_83c5_2ad7a7ba0a23.slice/crio-b0e8aa58a291344eba4b434afbce91e5b10cca2bbebb96317d4229d31448c4ca WatchSource:0}: Error finding container b0e8aa58a291344eba4b434afbce91e5b10cca2bbebb96317d4229d31448c4ca: Status 404 returned error can't find the container with id b0e8aa58a291344eba4b434afbce91e5b10cca2bbebb96317d4229d31448c4ca Nov 02 20:43:49.290553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.290532 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6"] Nov 02 20:43:49.292443 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.292415 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:43:49.292515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.292476 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:43:49.292515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.292506 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:43:49.299231 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.299210 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-kube-api-access-gh5vh\" (UniqueName: \"kubernetes.io/projected/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-kube-api-access-gh5vh\") pod \"3c0b5b7d-6e10-4dea-80df-69f60f219ff7\" (UID: \"3c0b5b7d-6e10-4dea-80df-69f60f219ff7\") " Nov 02 20:43:49.299335 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.299320 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c278625-8b76-4f43-95a3-510cb72ea044-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c278625-8b76-4f43-95a3-510cb72ea044-maas-api-xks-ca\") pod \"1c278625-8b76-4f43-95a3-510cb72ea044\" (UID: \"1c278625-8b76-4f43-95a3-510cb72ea044\") " Nov 02 20:43:49.299442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.299424 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c278625-8b76-4f43-95a3-510cb72ea044-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c278625-8b76-4f43-95a3-510cb72ea044-maas-api-openshift-ca\") pod \"1c278625-8b76-4f43-95a3-510cb72ea044\" (UID: \"1c278625-8b76-4f43-95a3-510cb72ea044\") " Nov 02 20:43:49.299515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.299499 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-maas-api-openshift-ca\") pod \"3c0b5b7d-6e10-4dea-80df-69f60f219ff7\" (UID: \"3c0b5b7d-6e10-4dea-80df-69f60f219ff7\") " Nov 02 20:43:49.299557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.299540 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-maas-api-xks-ca\") pod \"3c0b5b7d-6e10-4dea-80df-69f60f219ff7\" (UID: \"3c0b5b7d-6e10-4dea-80df-69f60f219ff7\") " Nov 02 20:43:49.299647 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.299631 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c278625-8b76-4f43-95a3-510cb72ea044-kube-api-access-ffd55\" (UniqueName: \"kubernetes.io/projected/1c278625-8b76-4f43-95a3-510cb72ea044-kube-api-access-ffd55\") pod \"1c278625-8b76-4f43-95a3-510cb72ea044\" (UID: \"1c278625-8b76-4f43-95a3-510cb72ea044\") " Nov 02 20:43:49.299844 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.299817 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c278625-8b76-4f43-95a3-510cb72ea044-maas-api-openshift-ca" pod "1c278625-8b76-4f43-95a3-510cb72ea044" (UID: "1c278625-8b76-4f43-95a3-510cb72ea044"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:49.299971 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.299921 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-maas-api-openshift-ca" pod "3c0b5b7d-6e10-4dea-80df-69f60f219ff7" (UID: "3c0b5b7d-6e10-4dea-80df-69f60f219ff7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:49.300348 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.300317 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c278625-8b76-4f43-95a3-510cb72ea044-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:49.300426 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.300355 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:49.302006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.301983 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c278625-8b76-4f43-95a3-510cb72ea044-maas-api-xks-ca" pod "1c278625-8b76-4f43-95a3-510cb72ea044" (UID: "1c278625-8b76-4f43-95a3-510cb72ea044"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:49.302220 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.302199 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-kube-api-access-gh5vh" pod "3c0b5b7d-6e10-4dea-80df-69f60f219ff7" (UID: "3c0b5b7d-6e10-4dea-80df-69f60f219ff7"). InnerVolumeSpecName "kube-api-access-gh5vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:49.302634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.302612 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-maas-api-xks-ca" pod "3c0b5b7d-6e10-4dea-80df-69f60f219ff7" (UID: "3c0b5b7d-6e10-4dea-80df-69f60f219ff7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:49.302634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.302620 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c278625-8b76-4f43-95a3-510cb72ea044-kube-api-access-ffd55" pod "1c278625-8b76-4f43-95a3-510cb72ea044" (UID: "1c278625-8b76-4f43-95a3-510cb72ea044"). InnerVolumeSpecName "kube-api-access-ffd55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:49.381336 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.381303 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q"] Nov 02 20:43:49.381577 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.381555 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerName="istio-proxy" containerID="cri-o://9dfcdf49d8bd7a38bf26c5ebb23536c4f65b58c59775bde471bdc699f723866e" gracePeriod=30 Nov 02 20:43:49.386039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.386013 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9"] Nov 02 20:43:49.386354 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.386325 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" podUID="b590ab7b-774d-4cce-9cc4-c15ebbe5181b" containerName="istio-proxy" containerID="cri-o://9a16382233ff47acdb32733847e899a5e125dce52a8c6d3bc94976e17b8c8544" gracePeriod=30 Nov 02 20:43:49.392159 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.392135 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v"] Nov 02 20:43:49.392427 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.392313 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" podUID="fa9309a3-38ea-4870-91dc-2bbab6b13f34" containerName="istio-proxy" containerID="cri-o://09e05c5dcf8c3580f416e3fbcd96335e567cc6a1f78c4d2aa2e2913c893af1df" gracePeriod=30 Nov 02 20:43:49.397456 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.397427 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5"] Nov 02 20:43:49.397643 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.397622 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" containerID="cri-o://6b15df5f0d893ce8967faef193df7e723cd273fd027ffc6d97e0e70d3ea7aa6f" gracePeriod=30 Nov 02 20:43:49.402289 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.402260 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh5vh\" (UniqueName: \"kubernetes.io/projected/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-kube-api-access-gh5vh\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:49.402289 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.402287 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c278625-8b76-4f43-95a3-510cb72ea044-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:49.402442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.402299 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0b5b7d-6e10-4dea-80df-69f60f219ff7-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:49.402442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.402307 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffd55\" (UniqueName: \"kubernetes.io/projected/1c278625-8b76-4f43-95a3-510cb72ea044-kube-api-access-ffd55\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:49.868483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.868451 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e" path="/var/lib/kubelet/pods/7f0ce06d-c8f9-49b8-8f7f-8b5fd2055c3e/volumes" Nov 02 20:43:49.984856 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.984829 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" Nov 02 20:43:49.984986 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.984867 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh" event={"ID":"1c278625-8b76-4f43-95a3-510cb72ea044","Type":"ContainerDied","Data":"d27316e9a50baae4fbb643f0c93d9e81faf5b26d582f439973f48885163b7e81"} Nov 02 20:43:49.984986 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.984920 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d27316e9a50baae4fbb643f0c93d9e81faf5b26d582f439973f48885163b7e81" Nov 02 20:43:49.986656 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.986608 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" Nov 02 20:43:49.986656 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.986627 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m" event={"ID":"3c0b5b7d-6e10-4dea-80df-69f60f219ff7","Type":"ContainerDied","Data":"23c4f1176fd567e7d1dc6511cfd86e0d3dec56e821c8f735e84f4ca5c74b4b78"} Nov 02 20:43:49.986952 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.986666 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23c4f1176fd567e7d1dc6511cfd86e0d3dec56e821c8f735e84f4ca5c74b4b78" Nov 02 20:43:49.988892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.988737 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" event={"ID":"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23","Type":"ContainerStarted","Data":"541c576ec9025f95e2e791888e95539786b459018e8def278a153f7ac0008cc5"} Nov 02 20:43:49.988892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:49.988782 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" event={"ID":"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23","Type":"ContainerStarted","Data":"b0e8aa58a291344eba4b434afbce91e5b10cca2bbebb96317d4229d31448c4ca"} Nov 02 20:43:50.105337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:50.105294 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" podStartSLOduration=2.105281103 podStartE2EDuration="2.105281103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:43:50.102205623 +0000 UTC m=+1758.391862150" watchObservedRunningTime="2026-11-02 20:43:50.105281103 +0000 UTC m=+1758.394937631" Nov 02 20:43:50.797761 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:50.797732 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:50.803379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:50.803350 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:50.996902 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:50.996875 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:50.997930 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:50.997902 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:43:51.578455 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:51.578401 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:51.835137 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:51.835020 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:51.858144 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:51.857186 4971 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:43:51.858144 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:51.857271 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" podUID="b590ab7b-774d-4cce-9cc4-c15ebbe5181b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:43:52.076623 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:52.076582 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:52.324803 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:52.324772 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:53.166682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:53.166652 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:53.414339 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:53.414307 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:53.550651 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:53.550577 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n"] Nov 02 20:43:53.550810 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:53.550786 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" podUID="191d909c-208a-43e3-91ad-9d0f979b1cc8" containerName="istio-proxy" containerID="cri-o://dc22361ee09fb630cbea7d87d01224409eb15da98838999d775c65ad11d51306" gracePeriod=30 Nov 02 20:43:54.712247 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.712225 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:43:54.716822 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.716797 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:43:54.759748 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.759727 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:43:54.810948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.810920 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-podinfo\") pod \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " Nov 02 20:43:54.810948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.810957 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-kube-api-access-rsghx\" (UniqueName: \"kubernetes.io/projected/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-kube-api-access-rsghx\") pod \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " Nov 02 20:43:54.811183 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811005 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-podinfo\") pod \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " Nov 02 20:43:54.811183 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811066 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-token\" (UniqueName: \"kubernetes.io/projected/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-token\") pod \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " Nov 02 20:43:54.811183 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811142 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa9309a3-38ea-4870-91dc-2bbab6b13f34-kube-api-access-xdmf6\" (UniqueName: \"kubernetes.io/projected/fa9309a3-38ea-4870-91dc-2bbab6b13f34-kube-api-access-xdmf6\") pod \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " Nov 02 20:43:54.811183 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811177 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-workload-socket\") pod \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " Nov 02 20:43:54.811428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811208 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-data\") pod \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " Nov 02 20:43:54.811428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811254 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-data\") pod \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " Nov 02 20:43:54.811428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811308 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istiod-ca-cert\") pod \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " Nov 02 20:43:54.811428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811340 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-token\" (UniqueName: \"kubernetes.io/projected/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-token\") pod \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " Nov 02 20:43:54.811428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811419 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-envoy\") pod \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " Nov 02 20:43:54.811682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811484 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-workload-socket\") pod \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " Nov 02 20:43:54.811682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811519 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istiod-ca-cert\") pod \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " Nov 02 20:43:54.811682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811570 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-workload-certs\") pod \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " Nov 02 20:43:54.811682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811605 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-credential-socket\") pod \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\" (UID: \"b590ab7b-774d-4cce-9cc4-c15ebbe5181b\") " Nov 02 20:43:54.811682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811632 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-credential-socket\") pod \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " Nov 02 20:43:54.811682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811668 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-envoy\") pod \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " Nov 02 20:43:54.812057 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811864 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-workload-certs\") pod \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " Nov 02 20:43:54.812057 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811871 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-data" pod "75f18eb1-5617-4f8a-b812-b0b9abe2078e" (UID: "75f18eb1-5617-4f8a-b812-b0b9abe2078e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.812057 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811903 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-token\" (UniqueName: \"kubernetes.io/projected/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-token\") pod \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " Nov 02 20:43:54.812057 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811937 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-podinfo\") pod \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " Nov 02 20:43:54.812057 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.811976 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-workload-socket\") pod \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " Nov 02 20:43:54.812057 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.812029 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-envoy\") pod \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " Nov 02 20:43:54.812402 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.812067 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-workload-certs\") pod \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " Nov 02 20:43:54.812402 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.812151 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istiod-ca-cert\") pod \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " Nov 02 20:43:54.812402 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.812175 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-workload-socket" pod "b590ab7b-774d-4cce-9cc4-c15ebbe5181b" (UID: "b590ab7b-774d-4cce-9cc4-c15ebbe5181b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.812402 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.812175 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istiod-ca-cert" pod "75f18eb1-5617-4f8a-b812-b0b9abe2078e" (UID: "75f18eb1-5617-4f8a-b812-b0b9abe2078e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:54.812402 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.812292 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f18eb1-5617-4f8a-b812-b0b9abe2078e-kube-api-access-cxkdv\" (UniqueName: \"kubernetes.io/projected/75f18eb1-5617-4f8a-b812-b0b9abe2078e-kube-api-access-cxkdv\") pod \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " Nov 02 20:43:54.812402 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.812322 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-data\") pod \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\" (UID: \"fa9309a3-38ea-4870-91dc-2bbab6b13f34\") " Nov 02 20:43:54.812402 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.812358 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-credential-socket\") pod \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\" (UID: \"75f18eb1-5617-4f8a-b812-b0b9abe2078e\") " Nov 02 20:43:54.812402 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.812371 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-workload-certs" pod "75f18eb1-5617-4f8a-b812-b0b9abe2078e" (UID: "75f18eb1-5617-4f8a-b812-b0b9abe2078e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.812819 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.812502 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-data" pod "b590ab7b-774d-4cce-9cc4-c15ebbe5181b" (UID: "b590ab7b-774d-4cce-9cc4-c15ebbe5181b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.812819 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.812543 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-workload-socket" pod "75f18eb1-5617-4f8a-b812-b0b9abe2078e" (UID: "75f18eb1-5617-4f8a-b812-b0b9abe2078e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.812819 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.812769 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-credential-socket" pod "75f18eb1-5617-4f8a-b812-b0b9abe2078e" (UID: "75f18eb1-5617-4f8a-b812-b0b9abe2078e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.814404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.813104 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istiod-ca-cert" pod "b590ab7b-774d-4cce-9cc4-c15ebbe5181b" (UID: "b590ab7b-774d-4cce-9cc4-c15ebbe5181b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:54.814404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.813336 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-workload-certs" pod "fa9309a3-38ea-4870-91dc-2bbab6b13f34" (UID: "fa9309a3-38ea-4870-91dc-2bbab6b13f34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.814404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.813401 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.814404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.813421 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.814404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.813437 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.814404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.813451 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.814404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.813465 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.814404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.813479 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.814404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.813492 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.814404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.813508 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.814404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.813618 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istiod-ca-cert" pod "fa9309a3-38ea-4870-91dc-2bbab6b13f34" (UID: "fa9309a3-38ea-4870-91dc-2bbab6b13f34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:54.814404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.813729 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-workload-certs" pod "b590ab7b-774d-4cce-9cc4-c15ebbe5181b" (UID: "b590ab7b-774d-4cce-9cc4-c15ebbe5181b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.814404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.813933 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-credential-socket" pod "b590ab7b-774d-4cce-9cc4-c15ebbe5181b" (UID: "b590ab7b-774d-4cce-9cc4-c15ebbe5181b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.814404 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.814142 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-credential-socket" pod "fa9309a3-38ea-4870-91dc-2bbab6b13f34" (UID: "fa9309a3-38ea-4870-91dc-2bbab6b13f34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.816149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.816051 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-token" pod "75f18eb1-5617-4f8a-b812-b0b9abe2078e" (UID: "75f18eb1-5617-4f8a-b812-b0b9abe2078e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:54.816321 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.816297 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-workload-socket" pod "fa9309a3-38ea-4870-91dc-2bbab6b13f34" (UID: "fa9309a3-38ea-4870-91dc-2bbab6b13f34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.816401 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.816348 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f18eb1-5617-4f8a-b812-b0b9abe2078e-kube-api-access-cxkdv" pod "75f18eb1-5617-4f8a-b812-b0b9abe2078e" (UID: "75f18eb1-5617-4f8a-b812-b0b9abe2078e"). InnerVolumeSpecName "kube-api-access-cxkdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:54.816595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.816570 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-data" pod "fa9309a3-38ea-4870-91dc-2bbab6b13f34" (UID: "fa9309a3-38ea-4870-91dc-2bbab6b13f34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.817217 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.817161 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-podinfo" pod "fa9309a3-38ea-4870-91dc-2bbab6b13f34" (UID: "fa9309a3-38ea-4870-91dc-2bbab6b13f34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:54.817648 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.817622 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-token" pod "fa9309a3-38ea-4870-91dc-2bbab6b13f34" (UID: "fa9309a3-38ea-4870-91dc-2bbab6b13f34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:54.818638 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.818509 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-kube-api-access-rsghx" pod "b590ab7b-774d-4cce-9cc4-c15ebbe5181b" (UID: "b590ab7b-774d-4cce-9cc4-c15ebbe5181b"). InnerVolumeSpecName "kube-api-access-rsghx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:54.818736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.818715 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-token" pod "b590ab7b-774d-4cce-9cc4-c15ebbe5181b" (UID: "b590ab7b-774d-4cce-9cc4-c15ebbe5181b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:54.819018 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.818995 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-envoy" pod "fa9309a3-38ea-4870-91dc-2bbab6b13f34" (UID: "fa9309a3-38ea-4870-91dc-2bbab6b13f34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.819799 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.819767 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9309a3-38ea-4870-91dc-2bbab6b13f34-kube-api-access-xdmf6" pod "fa9309a3-38ea-4870-91dc-2bbab6b13f34" (UID: "fa9309a3-38ea-4870-91dc-2bbab6b13f34"). InnerVolumeSpecName "kube-api-access-xdmf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:54.819935 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.819916 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-envoy" pod "b590ab7b-774d-4cce-9cc4-c15ebbe5181b" (UID: "b590ab7b-774d-4cce-9cc4-c15ebbe5181b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.820060 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.820043 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-podinfo" pod "75f18eb1-5617-4f8a-b812-b0b9abe2078e" (UID: "75f18eb1-5617-4f8a-b812-b0b9abe2078e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:54.820387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.820366 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-podinfo" pod "b590ab7b-774d-4cce-9cc4-c15ebbe5181b" (UID: "b590ab7b-774d-4cce-9cc4-c15ebbe5181b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:54.821614 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.821586 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-envoy" pod "75f18eb1-5617-4f8a-b812-b0b9abe2078e" (UID: "75f18eb1-5617-4f8a-b812-b0b9abe2078e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914733 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914769 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914783 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914798 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914813 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914826 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914840 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914854 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxkdv\" (UniqueName: \"kubernetes.io/projected/75f18eb1-5617-4f8a-b812-b0b9abe2078e-kube-api-access-cxkdv\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914868 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914881 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914900 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsghx\" (UniqueName: \"kubernetes.io/projected/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-kube-api-access-rsghx\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914917 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914933 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914951 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdmf6\" (UniqueName: \"kubernetes.io/projected/fa9309a3-38ea-4870-91dc-2bbab6b13f34-kube-api-access-xdmf6\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914965 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f18eb1-5617-4f8a-b812-b0b9abe2078e-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914978 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.914991 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.915004 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7b-774d-4cce-9cc4-c15ebbe5181b-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.915029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:54.915012 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa9309a3-38ea-4870-91dc-2bbab6b13f34-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:55.050724 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.050692 4971 generic.go:356] "Generic (PLEG): container finished" podID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerID="9dfcdf49d8bd7a38bf26c5ebb23536c4f65b58c59775bde471bdc699f723866e" exitCode=0 Nov 02 20:43:55.050872 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.050763 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" event={"ID":"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50","Type":"ContainerDied","Data":"9dfcdf49d8bd7a38bf26c5ebb23536c4f65b58c59775bde471bdc699f723866e"} Nov 02 20:43:55.052459 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.052431 4971 generic.go:356] "Generic (PLEG): container finished" podID="fa9309a3-38ea-4870-91dc-2bbab6b13f34" containerID="09e05c5dcf8c3580f416e3fbcd96335e567cc6a1f78c4d2aa2e2913c893af1df" exitCode=0 Nov 02 20:43:55.052618 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.052600 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" event={"ID":"fa9309a3-38ea-4870-91dc-2bbab6b13f34","Type":"ContainerDied","Data":"09e05c5dcf8c3580f416e3fbcd96335e567cc6a1f78c4d2aa2e2913c893af1df"} Nov 02 20:43:55.052660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.052639 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" event={"ID":"fa9309a3-38ea-4870-91dc-2bbab6b13f34","Type":"ContainerDied","Data":"9e842f117767348354ec13bd864a66434708ceaa5362b3ac3bad4727e59ee47f"} Nov 02 20:43:55.052693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.052661 4971 scope.go:122] "RemoveContainer" containerID="09e05c5dcf8c3580f416e3fbcd96335e567cc6a1f78c4d2aa2e2913c893af1df" Nov 02 20:43:55.052791 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.052781 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v" Nov 02 20:43:55.060324 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.060300 4971 generic.go:356] "Generic (PLEG): container finished" podID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerID="6b15df5f0d893ce8967faef193df7e723cd273fd027ffc6d97e0e70d3ea7aa6f" exitCode=0 Nov 02 20:43:55.061215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.060534 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" Nov 02 20:43:55.061215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.060392 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" event={"ID":"75f18eb1-5617-4f8a-b812-b0b9abe2078e","Type":"ContainerDied","Data":"6b15df5f0d893ce8967faef193df7e723cd273fd027ffc6d97e0e70d3ea7aa6f"} Nov 02 20:43:55.061215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.060653 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5" event={"ID":"75f18eb1-5617-4f8a-b812-b0b9abe2078e","Type":"ContainerDied","Data":"04df6782884d84390ecff7f2b76d7f543f68a9ca823068202d198b2f859f18d8"} Nov 02 20:43:55.063776 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.063752 4971 generic.go:356] "Generic (PLEG): container finished" podID="b590ab7b-774d-4cce-9cc4-c15ebbe5181b" containerID="9a16382233ff47acdb32733847e899a5e125dce52a8c6d3bc94976e17b8c8544" exitCode=0 Nov 02 20:43:55.063852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.063786 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" event={"ID":"b590ab7b-774d-4cce-9cc4-c15ebbe5181b","Type":"ContainerDied","Data":"9a16382233ff47acdb32733847e899a5e125dce52a8c6d3bc94976e17b8c8544"} Nov 02 20:43:55.063852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.063815 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" Nov 02 20:43:55.063852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.063826 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9" event={"ID":"b590ab7b-774d-4cce-9cc4-c15ebbe5181b","Type":"ContainerDied","Data":"2a72b284bafb94413cf700bf85dc10537972189f4bbbaa238bd846fc9f76cb7a"} Nov 02 20:43:55.074280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.074260 4971 scope.go:122] "RemoveContainer" containerID="09e05c5dcf8c3580f416e3fbcd96335e567cc6a1f78c4d2aa2e2913c893af1df" Nov 02 20:43:55.074612 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:43:55.074589 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e05c5dcf8c3580f416e3fbcd96335e567cc6a1f78c4d2aa2e2913c893af1df\": container with ID starting with 09e05c5dcf8c3580f416e3fbcd96335e567cc6a1f78c4d2aa2e2913c893af1df not found: ID does not exist" containerID="09e05c5dcf8c3580f416e3fbcd96335e567cc6a1f78c4d2aa2e2913c893af1df" Nov 02 20:43:55.074746 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.074621 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e05c5dcf8c3580f416e3fbcd96335e567cc6a1f78c4d2aa2e2913c893af1df"} err="failed to get container status \"09e05c5dcf8c3580f416e3fbcd96335e567cc6a1f78c4d2aa2e2913c893af1df\": rpc error: code = NotFound desc = could not find container \"09e05c5dcf8c3580f416e3fbcd96335e567cc6a1f78c4d2aa2e2913c893af1df\": container with ID starting with 09e05c5dcf8c3580f416e3fbcd96335e567cc6a1f78c4d2aa2e2913c893af1df not found: ID does not exist" Nov 02 20:43:55.074746 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.074644 4971 scope.go:122] "RemoveContainer" containerID="6b15df5f0d893ce8967faef193df7e723cd273fd027ffc6d97e0e70d3ea7aa6f" Nov 02 20:43:55.094347 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.094321 4971 scope.go:122] "RemoveContainer" containerID="6b15df5f0d893ce8967faef193df7e723cd273fd027ffc6d97e0e70d3ea7aa6f" Nov 02 20:43:55.094640 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:43:55.094622 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b15df5f0d893ce8967faef193df7e723cd273fd027ffc6d97e0e70d3ea7aa6f\": container with ID starting with 6b15df5f0d893ce8967faef193df7e723cd273fd027ffc6d97e0e70d3ea7aa6f not found: ID does not exist" containerID="6b15df5f0d893ce8967faef193df7e723cd273fd027ffc6d97e0e70d3ea7aa6f" Nov 02 20:43:55.094691 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.094648 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b15df5f0d893ce8967faef193df7e723cd273fd027ffc6d97e0e70d3ea7aa6f"} err="failed to get container status \"6b15df5f0d893ce8967faef193df7e723cd273fd027ffc6d97e0e70d3ea7aa6f\": rpc error: code = NotFound desc = could not find container \"6b15df5f0d893ce8967faef193df7e723cd273fd027ffc6d97e0e70d3ea7aa6f\": container with ID starting with 6b15df5f0d893ce8967faef193df7e723cd273fd027ffc6d97e0e70d3ea7aa6f not found: ID does not exist" Nov 02 20:43:55.094691 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.094664 4971 scope.go:122] "RemoveContainer" containerID="9a16382233ff47acdb32733847e899a5e125dce52a8c6d3bc94976e17b8c8544" Nov 02 20:43:55.105144 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.105130 4971 scope.go:122] "RemoveContainer" containerID="9a16382233ff47acdb32733847e899a5e125dce52a8c6d3bc94976e17b8c8544" Nov 02 20:43:55.105364 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:43:55.105344 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a16382233ff47acdb32733847e899a5e125dce52a8c6d3bc94976e17b8c8544\": container with ID starting with 9a16382233ff47acdb32733847e899a5e125dce52a8c6d3bc94976e17b8c8544 not found: ID does not exist" containerID="9a16382233ff47acdb32733847e899a5e125dce52a8c6d3bc94976e17b8c8544" Nov 02 20:43:55.105417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.105372 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a16382233ff47acdb32733847e899a5e125dce52a8c6d3bc94976e17b8c8544"} err="failed to get container status \"9a16382233ff47acdb32733847e899a5e125dce52a8c6d3bc94976e17b8c8544\": rpc error: code = NotFound desc = could not find container \"9a16382233ff47acdb32733847e899a5e125dce52a8c6d3bc94976e17b8c8544\": container with ID starting with 9a16382233ff47acdb32733847e899a5e125dce52a8c6d3bc94976e17b8c8544 not found: ID does not exist" Nov 02 20:43:55.110295 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.110146 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:43:55.222513 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.222427 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-workload-socket\") pod \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " Nov 02 20:43:55.222513 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.222491 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-workload-certs\") pod \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " Nov 02 20:43:55.222769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.222538 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-envoy\") pod \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " Nov 02 20:43:55.222769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.222572 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-kube-api-access-2gspl\" (UniqueName: \"kubernetes.io/projected/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-kube-api-access-2gspl\") pod \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " Nov 02 20:43:55.222769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.222708 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-workload-socket" pod "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" (UID: "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:55.222769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.222733 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-token\" (UniqueName: \"kubernetes.io/projected/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-token\") pod \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.222808 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-podinfo\") pod \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.222879 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-workload-certs" pod "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" (UID: "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.222934 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-data\") pod \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.223287 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-data" pod "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" (UID: "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.223396 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istiod-ca-cert\") pod \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.223489 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-credential-socket\") pod \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\" (UID: \"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50\") " Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.224159 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istiod-ca-cert" pod "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" (UID: "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.224549 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-credential-socket" pod "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" (UID: "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.228815 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.228862 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.228872 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.228926 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.228938 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.230648 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-kube-api-access-2gspl" pod "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" (UID: "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50"). InnerVolumeSpecName "kube-api-access-2gspl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.231021 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-podinfo" pod "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" (UID: "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.231976 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9"] Nov 02 20:43:55.235278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.233624 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-envoy" pod "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" (UID: "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:55.236035 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.235713 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-token" pod "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" (UID: "a416fb5c-b8c6-4159-9cc3-d2f6ea601b50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:55.237757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.237738 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-pnzm9"] Nov 02 20:43:55.329883 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.329854 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:55.329883 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.329879 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gspl\" (UniqueName: \"kubernetes.io/projected/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-kube-api-access-2gspl\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:55.330058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.329893 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:55.330058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.329907 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:55.331951 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.331927 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v"] Nov 02 20:43:55.336999 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.336976 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-sgs7v"] Nov 02 20:43:55.565787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.565689 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5"] Nov 02 20:43:55.570864 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.570840 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-r2jw5"] Nov 02 20:43:55.868220 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.868192 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" path="/var/lib/kubelet/pods/75f18eb1-5617-4f8a-b812-b0b9abe2078e/volumes" Nov 02 20:43:55.868693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.868677 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b590ab7b-774d-4cce-9cc4-c15ebbe5181b" path="/var/lib/kubelet/pods/b590ab7b-774d-4cce-9cc4-c15ebbe5181b/volumes" Nov 02 20:43:55.869172 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:55.869154 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa9309a3-38ea-4870-91dc-2bbab6b13f34" path="/var/lib/kubelet/pods/fa9309a3-38ea-4870-91dc-2bbab6b13f34/volumes" Nov 02 20:43:56.076138 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:56.076107 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" Nov 02 20:43:56.076269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:56.076105 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q" event={"ID":"a416fb5c-b8c6-4159-9cc3-d2f6ea601b50","Type":"ContainerDied","Data":"4261490399a8a6f854146b881dc4ac7f177290e20bb2d8ba7afd74efdb422280"} Nov 02 20:43:56.076269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:56.076232 4971 scope.go:122] "RemoveContainer" containerID="9dfcdf49d8bd7a38bf26c5ebb23536c4f65b58c59775bde471bdc699f723866e" Nov 02 20:43:56.235520 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:56.235457 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q"] Nov 02 20:43:56.283715 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:56.283686 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-kgn5q"] Nov 02 20:43:56.469204 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:56.466714 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:56.724916 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:56.724335 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.103608 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq"] Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104441 4971 cpu_manager.go:414] "containerMap: removing container" podUID="fa9309a3-38ea-4870-91dc-2bbab6b13f34" containerName="istio-proxy" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104465 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa9309a3-38ea-4870-91dc-2bbab6b13f34" containerName="istio-proxy" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104479 4971 cpu_manager.go:414] "containerMap: removing container" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerName="istio-proxy" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104487 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerName="istio-proxy" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104545 4971 cpu_manager.go:414] "containerMap: removing container" podUID="b590ab7b-774d-4cce-9cc4-c15ebbe5181b" containerName="istio-proxy" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104554 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b590ab7b-774d-4cce-9cc4-c15ebbe5181b" containerName="istio-proxy" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104568 4971 cpu_manager.go:414] "containerMap: removing container" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104575 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104606 4971 cpu_manager.go:414] "containerMap: removing container" podUID="3c0b5b7d-6e10-4dea-80df-69f60f219ff7" containerName="revoke-keys" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104613 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0b5b7d-6e10-4dea-80df-69f60f219ff7" containerName="revoke-keys" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104632 4971 cpu_manager.go:414] "containerMap: removing container" podUID="1c278625-8b76-4f43-95a3-510cb72ea044" containerName="revoke-keys" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104638 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c278625-8b76-4f43-95a3-510cb72ea044" containerName="revoke-keys" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104810 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0b5b7d-6e10-4dea-80df-69f60f219ff7" containerName="revoke-keys" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104823 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c278625-8b76-4f43-95a3-510cb72ea044" containerName="revoke-keys" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104834 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa9309a3-38ea-4870-91dc-2bbab6b13f34" containerName="istio-proxy" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104846 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" containerName="istio-proxy" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104863 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="b590ab7b-774d-4cce-9cc4-c15ebbe5181b" containerName="istio-proxy" Nov 02 20:43:57.105092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.104874 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f18eb1-5617-4f8a-b812-b0b9abe2078e" containerName="istio-proxy" Nov 02 20:43:57.114160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.111548 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" Nov 02 20:43:57.157024 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.156997 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2f778f-16a7-45fd-912a-fe2098795d7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq\" (UID: \"7e2f778f-16a7-45fd-912a-fe2098795d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" Nov 02 20:43:57.157193 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.157103 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tj5n\" (UniqueName: \"kubernetes.io/projected/7e2f778f-16a7-45fd-912a-fe2098795d7c-kube-api-access-2tj5n\") pod \"payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq\" (UID: \"7e2f778f-16a7-45fd-912a-fe2098795d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" Nov 02 20:43:57.158276 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.158250 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq"] Nov 02 20:43:57.164202 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.164151 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-039b89-dockercfg-wrkmf\"" Nov 02 20:43:57.172572 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.172553 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-039b89\"" Nov 02 20:43:57.258023 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.257987 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2f778f-16a7-45fd-912a-fe2098795d7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq\" (UID: \"7e2f778f-16a7-45fd-912a-fe2098795d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" Nov 02 20:43:57.258238 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.258064 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tj5n\" (UniqueName: \"kubernetes.io/projected/7e2f778f-16a7-45fd-912a-fe2098795d7c-kube-api-access-2tj5n\") pod \"payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq\" (UID: \"7e2f778f-16a7-45fd-912a-fe2098795d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" Nov 02 20:43:57.258783 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.258755 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2f778f-16a7-45fd-912a-fe2098795d7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq\" (UID: \"7e2f778f-16a7-45fd-912a-fe2098795d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" Nov 02 20:43:57.317017 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.316989 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tj5n\" (UniqueName: \"kubernetes.io/projected/7e2f778f-16a7-45fd-912a-fe2098795d7c-kube-api-access-2tj5n\") pod \"payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq\" (UID: \"7e2f778f-16a7-45fd-912a-fe2098795d7c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" Nov 02 20:43:57.427338 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.427256 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" Nov 02 20:43:57.437119 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.437053 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:57.811549 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:43:57.811520 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2f778f_16a7_45fd_912a_fe2098795d7c.slice/crio-2c85efcc56c4e3ea7af85986fcee031bb08fe081e267f00d01b1fd7aa7d6b4b3 WatchSource:0}: Error finding container 2c85efcc56c4e3ea7af85986fcee031bb08fe081e267f00d01b1fd7aa7d6b4b3: Status 404 returned error can't find the container with id 2c85efcc56c4e3ea7af85986fcee031bb08fe081e267f00d01b1fd7aa7d6b4b3 Nov 02 20:43:57.812676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.812653 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq"] Nov 02 20:43:57.868980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:57.868950 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a416fb5c-b8c6-4159-9cc3-d2f6ea601b50" path="/var/lib/kubelet/pods/a416fb5c-b8c6-4159-9cc3-d2f6ea601b50/volumes" Nov 02 20:43:58.123401 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.123366 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" event={"ID":"7e2f778f-16a7-45fd-912a-fe2098795d7c","Type":"ContainerStarted","Data":"a10c6446e4b5fa79051d5cb7ea1fee720b51f0ebde7e12036d14ef28b6ef8218"} Nov 02 20:43:58.123864 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.123430 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" event={"ID":"7e2f778f-16a7-45fd-912a-fe2098795d7c","Type":"ContainerStarted","Data":"2c85efcc56c4e3ea7af85986fcee031bb08fe081e267f00d01b1fd7aa7d6b4b3"} Nov 02 20:43:58.123864 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.123459 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" Nov 02 20:43:58.235755 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.235710 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" podStartSLOduration=1.235697189 podStartE2EDuration="1.235697189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:43:58.233292188 +0000 UTC m=+1766.522948730" watchObservedRunningTime="2026-11-02 20:43:58.235697189 +0000 UTC m=+1766.525353717" Nov 02 20:43:58.839328 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.839301 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:43:58.884750 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.884723 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-envoy\") pod \"191d909c-208a-43e3-91ad-9d0f979b1cc8\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " Nov 02 20:43:58.884891 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.884769 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-workload-certs\") pod \"191d909c-208a-43e3-91ad-9d0f979b1cc8\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " Nov 02 20:43:58.884891 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.884792 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-token\" (UniqueName: \"kubernetes.io/projected/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-token\") pod \"191d909c-208a-43e3-91ad-9d0f979b1cc8\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " Nov 02 20:43:58.884891 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.884821 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-credential-socket\") pod \"191d909c-208a-43e3-91ad-9d0f979b1cc8\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " Nov 02 20:43:58.884891 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.884852 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/191d909c-208a-43e3-91ad-9d0f979b1cc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191d909c-208a-43e3-91ad-9d0f979b1cc8-istiod-ca-cert\") pod \"191d909c-208a-43e3-91ad-9d0f979b1cc8\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " Nov 02 20:43:58.885129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.884925 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191d909c-208a-43e3-91ad-9d0f979b1cc8-kube-api-access-rptjf\" (UniqueName: \"kubernetes.io/projected/191d909c-208a-43e3-91ad-9d0f979b1cc8-kube-api-access-rptjf\") pod \"191d909c-208a-43e3-91ad-9d0f979b1cc8\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " Nov 02 20:43:58.885129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.885037 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-workload-certs" pod "191d909c-208a-43e3-91ad-9d0f979b1cc8" (UID: "191d909c-208a-43e3-91ad-9d0f979b1cc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:58.885129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.885061 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-podinfo\") pod \"191d909c-208a-43e3-91ad-9d0f979b1cc8\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " Nov 02 20:43:58.885315 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.885170 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-credential-socket" pod "191d909c-208a-43e3-91ad-9d0f979b1cc8" (UID: "191d909c-208a-43e3-91ad-9d0f979b1cc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:58.885315 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.885186 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-workload-socket\") pod \"191d909c-208a-43e3-91ad-9d0f979b1cc8\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " Nov 02 20:43:58.885315 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.885229 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-data\") pod \"191d909c-208a-43e3-91ad-9d0f979b1cc8\" (UID: \"191d909c-208a-43e3-91ad-9d0f979b1cc8\") " Nov 02 20:43:58.885483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.885395 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-workload-socket" pod "191d909c-208a-43e3-91ad-9d0f979b1cc8" (UID: "191d909c-208a-43e3-91ad-9d0f979b1cc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:58.885483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.885405 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191d909c-208a-43e3-91ad-9d0f979b1cc8-istiod-ca-cert" pod "191d909c-208a-43e3-91ad-9d0f979b1cc8" (UID: "191d909c-208a-43e3-91ad-9d0f979b1cc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:58.885627 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.885601 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-data" pod "191d909c-208a-43e3-91ad-9d0f979b1cc8" (UID: "191d909c-208a-43e3-91ad-9d0f979b1cc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:58.885960 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.885904 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:58.885960 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.885927 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:58.885960 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.885936 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:58.885960 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.885945 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:58.885960 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.885954 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191d909c-208a-43e3-91ad-9d0f979b1cc8-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:58.887269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.887249 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-token" pod "191d909c-208a-43e3-91ad-9d0f979b1cc8" (UID: "191d909c-208a-43e3-91ad-9d0f979b1cc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:58.887628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.887613 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191d909c-208a-43e3-91ad-9d0f979b1cc8-kube-api-access-rptjf" pod "191d909c-208a-43e3-91ad-9d0f979b1cc8" (UID: "191d909c-208a-43e3-91ad-9d0f979b1cc8"). InnerVolumeSpecName "kube-api-access-rptjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:58.887763 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.887745 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-envoy" pod "191d909c-208a-43e3-91ad-9d0f979b1cc8" (UID: "191d909c-208a-43e3-91ad-9d0f979b1cc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:58.887811 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.887763 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-podinfo" pod "191d909c-208a-43e3-91ad-9d0f979b1cc8" (UID: "191d909c-208a-43e3-91ad-9d0f979b1cc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:58.934809 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.934767 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:58.988253 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.988221 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:58.988253 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.988254 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:58.988441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.988268 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191d909c-208a-43e3-91ad-9d0f979b1cc8-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:58.988441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:58.988283 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rptjf\" (UniqueName: \"kubernetes.io/projected/191d909c-208a-43e3-91ad-9d0f979b1cc8-kube-api-access-rptjf\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.133971 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:59.133929 4971 generic.go:356] "Generic (PLEG): container finished" podID="191d909c-208a-43e3-91ad-9d0f979b1cc8" containerID="dc22361ee09fb630cbea7d87d01224409eb15da98838999d775c65ad11d51306" exitCode=0 Nov 02 20:43:59.134491 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:59.134049 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" Nov 02 20:43:59.134491 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:59.134065 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" event={"ID":"191d909c-208a-43e3-91ad-9d0f979b1cc8","Type":"ContainerDied","Data":"dc22361ee09fb630cbea7d87d01224409eb15da98838999d775c65ad11d51306"} Nov 02 20:43:59.134491 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:59.134136 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n" event={"ID":"191d909c-208a-43e3-91ad-9d0f979b1cc8","Type":"ContainerDied","Data":"84b802fb7f04b7ef4d7fa98ef99eba2db02c6be641eff57ea8d0267dc9a23afa"} Nov 02 20:43:59.134491 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:59.134161 4971 scope.go:122] "RemoveContainer" containerID="dc22361ee09fb630cbea7d87d01224409eb15da98838999d775c65ad11d51306" Nov 02 20:43:59.148428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:59.148405 4971 scope.go:122] "RemoveContainer" containerID="dc22361ee09fb630cbea7d87d01224409eb15da98838999d775c65ad11d51306" Nov 02 20:43:59.148730 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:43:59.148707 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc22361ee09fb630cbea7d87d01224409eb15da98838999d775c65ad11d51306\": container with ID starting with dc22361ee09fb630cbea7d87d01224409eb15da98838999d775c65ad11d51306 not found: ID does not exist" containerID="dc22361ee09fb630cbea7d87d01224409eb15da98838999d775c65ad11d51306" Nov 02 20:43:59.148827 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:59.148743 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc22361ee09fb630cbea7d87d01224409eb15da98838999d775c65ad11d51306"} err="failed to get container status \"dc22361ee09fb630cbea7d87d01224409eb15da98838999d775c65ad11d51306\": rpc error: code = NotFound desc = could not find container \"dc22361ee09fb630cbea7d87d01224409eb15da98838999d775c65ad11d51306\": container with ID starting with dc22361ee09fb630cbea7d87d01224409eb15da98838999d775c65ad11d51306 not found: ID does not exist" Nov 02 20:43:59.181738 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:59.181716 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:43:59.297143 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:59.297110 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n"] Nov 02 20:43:59.302545 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:59.302518 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-v7b6n"] Nov 02 20:43:59.867519 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:43:59.867486 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191d909c-208a-43e3-91ad-9d0f979b1cc8" path="/var/lib/kubelet/pods/191d909c-208a-43e3-91ad-9d0f979b1cc8/volumes" Nov 02 20:44:02.713163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:02.713127 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:02.963288 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:02.963072 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:03.209148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:03.209105 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:03.340039 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:44:03.339951 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:44:03.861959 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:03.861926 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:04.320320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:04.320239 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:07.475775 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:07.475740 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:08.663988 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:08.663957 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd"] Nov 02 20:44:08.664427 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:08.664185 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" podUID="61a53826-3ab4-4bc6-a88e-dcb48addd13b" containerName="istio-proxy" containerID="cri-o://47c0ab726a8bd644c3ac10e236f633d5ab449d2dd1a3e942b76ec39b057b7922" gracePeriod=30 Nov 02 20:44:09.137537 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:09.137511 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" Nov 02 20:44:09.165260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:09.165235 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:09.446622 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:09.446518 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:09.566107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:09.566066 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc"] Nov 02 20:44:09.569861 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:09.569837 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3d8f2f-62hpc"] Nov 02 20:44:09.701311 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:09.701237 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:09.867880 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:09.867847 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1af677d-851c-4d0a-a4ae-ee7fa6c3f974" path="/var/lib/kubelet/pods/e1af677d-851c-4d0a-a4ae-ee7fa6c3f974/volumes" Nov 02 20:44:13.952255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:13.952189 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:44:14.040039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.040002 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-podinfo\") pod \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " Nov 02 20:44:14.040206 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.040186 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-credential-socket\") pod \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " Nov 02 20:44:14.040311 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.040245 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-envoy\") pod \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " Nov 02 20:44:14.040311 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.040321 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-token\" (UniqueName: \"kubernetes.io/projected/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-token\") pod \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " Nov 02 20:44:14.040452 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.040355 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-workload-socket\") pod \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " Nov 02 20:44:14.040452 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.040392 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-workload-certs\") pod \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " Nov 02 20:44:14.040452 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.040432 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-data\") pod \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " Nov 02 20:44:14.040609 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.040466 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a53826-3ab4-4bc6-a88e-dcb48addd13b-kube-api-access-2f9k9\" (UniqueName: \"kubernetes.io/projected/61a53826-3ab4-4bc6-a88e-dcb48addd13b-kube-api-access-2f9k9\") pod \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " Nov 02 20:44:14.040609 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.040530 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istiod-ca-cert\") pod \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\" (UID: \"61a53826-3ab4-4bc6-a88e-dcb48addd13b\") " Nov 02 20:44:14.040720 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.040683 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-workload-certs" pod "61a53826-3ab4-4bc6-a88e-dcb48addd13b" (UID: "61a53826-3ab4-4bc6-a88e-dcb48addd13b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:14.040720 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.040663 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-workload-socket" pod "61a53826-3ab4-4bc6-a88e-dcb48addd13b" (UID: "61a53826-3ab4-4bc6-a88e-dcb48addd13b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:14.040824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.040719 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-data" pod "61a53826-3ab4-4bc6-a88e-dcb48addd13b" (UID: "61a53826-3ab4-4bc6-a88e-dcb48addd13b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:14.040824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.040763 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-credential-socket" pod "61a53826-3ab4-4bc6-a88e-dcb48addd13b" (UID: "61a53826-3ab4-4bc6-a88e-dcb48addd13b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:14.041103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.041062 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istiod-ca-cert" pod "61a53826-3ab4-4bc6-a88e-dcb48addd13b" (UID: "61a53826-3ab4-4bc6-a88e-dcb48addd13b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:44:14.041375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.041318 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.041375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.041344 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.041375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.041353 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.041375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.041362 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.041375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.041371 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.042490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.042466 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-podinfo" pod "61a53826-3ab4-4bc6-a88e-dcb48addd13b" (UID: "61a53826-3ab4-4bc6-a88e-dcb48addd13b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:44:14.042898 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.042877 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a53826-3ab4-4bc6-a88e-dcb48addd13b-kube-api-access-2f9k9" pod "61a53826-3ab4-4bc6-a88e-dcb48addd13b" (UID: "61a53826-3ab4-4bc6-a88e-dcb48addd13b"). InnerVolumeSpecName "kube-api-access-2f9k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:14.043133 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.043111 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-token" pod "61a53826-3ab4-4bc6-a88e-dcb48addd13b" (UID: "61a53826-3ab4-4bc6-a88e-dcb48addd13b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:14.043207 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.043151 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-envoy" pod "61a53826-3ab4-4bc6-a88e-dcb48addd13b" (UID: "61a53826-3ab4-4bc6-a88e-dcb48addd13b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:14.142937 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.142889 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f9k9\" (UniqueName: \"kubernetes.io/projected/61a53826-3ab4-4bc6-a88e-dcb48addd13b-kube-api-access-2f9k9\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.142937 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.142930 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.143208 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.142948 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.143208 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.142963 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a53826-3ab4-4bc6-a88e-dcb48addd13b-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.303666 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.303572 4971 generic.go:356] "Generic (PLEG): container finished" podID="61a53826-3ab4-4bc6-a88e-dcb48addd13b" containerID="47c0ab726a8bd644c3ac10e236f633d5ab449d2dd1a3e942b76ec39b057b7922" exitCode=0 Nov 02 20:44:14.303666 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.303631 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" event={"ID":"61a53826-3ab4-4bc6-a88e-dcb48addd13b","Type":"ContainerDied","Data":"47c0ab726a8bd644c3ac10e236f633d5ab449d2dd1a3e942b76ec39b057b7922"} Nov 02 20:44:14.303666 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.303640 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" Nov 02 20:44:14.303963 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.303683 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd" event={"ID":"61a53826-3ab4-4bc6-a88e-dcb48addd13b","Type":"ContainerDied","Data":"be566058aac4acc33902e9b1a7e16b91220b5aab06d4ec3863679b42000fed53"} Nov 02 20:44:14.303963 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.303713 4971 scope.go:122] "RemoveContainer" containerID="47c0ab726a8bd644c3ac10e236f633d5ab449d2dd1a3e942b76ec39b057b7922" Nov 02 20:44:14.318758 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.318738 4971 scope.go:122] "RemoveContainer" containerID="47c0ab726a8bd644c3ac10e236f633d5ab449d2dd1a3e942b76ec39b057b7922" Nov 02 20:44:14.319094 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:44:14.319058 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c0ab726a8bd644c3ac10e236f633d5ab449d2dd1a3e942b76ec39b057b7922\": container with ID starting with 47c0ab726a8bd644c3ac10e236f633d5ab449d2dd1a3e942b76ec39b057b7922 not found: ID does not exist" containerID="47c0ab726a8bd644c3ac10e236f633d5ab449d2dd1a3e942b76ec39b057b7922" Nov 02 20:44:14.319167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.319126 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c0ab726a8bd644c3ac10e236f633d5ab449d2dd1a3e942b76ec39b057b7922"} err="failed to get container status \"47c0ab726a8bd644c3ac10e236f633d5ab449d2dd1a3e942b76ec39b057b7922\": rpc error: code = NotFound desc = could not find container \"47c0ab726a8bd644c3ac10e236f633d5ab449d2dd1a3e942b76ec39b057b7922\": container with ID starting with 47c0ab726a8bd644c3ac10e236f633d5ab449d2dd1a3e942b76ec39b057b7922 not found: ID does not exist" Nov 02 20:44:14.514214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.514182 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd"] Nov 02 20:44:14.519692 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.519637 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-548dd"] Nov 02 20:44:14.639305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:14.639259 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:15.456652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.456614 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc"] Nov 02 20:44:15.458255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.458215 4971 cpu_manager.go:414] "containerMap: removing container" podUID="191d909c-208a-43e3-91ad-9d0f979b1cc8" containerName="istio-proxy" Nov 02 20:44:15.458255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.458250 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191d909c-208a-43e3-91ad-9d0f979b1cc8" containerName="istio-proxy" Nov 02 20:44:15.458462 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.458270 4971 cpu_manager.go:414] "containerMap: removing container" podUID="61a53826-3ab4-4bc6-a88e-dcb48addd13b" containerName="istio-proxy" Nov 02 20:44:15.458462 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.458276 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a53826-3ab4-4bc6-a88e-dcb48addd13b" containerName="istio-proxy" Nov 02 20:44:15.458644 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.458628 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="191d909c-208a-43e3-91ad-9d0f979b1cc8" containerName="istio-proxy" Nov 02 20:44:15.458697 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.458649 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a53826-3ab4-4bc6-a88e-dcb48addd13b" containerName="istio-proxy" Nov 02 20:44:15.464191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.464168 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.517669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.517641 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc"] Nov 02 20:44:15.559096 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.559048 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-18fd4c-openshift-default-dockercfg-p6c5z\"" Nov 02 20:44:15.559936 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.559903 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-workload-socket\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.560076 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.559981 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5fqt\" (UniqueName: \"kubernetes.io/projected/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-kube-api-access-k5fqt\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.560076 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.560021 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-workload-certs\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.560315 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.560217 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istiod-ca-cert\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.560382 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.560321 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-podinfo\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.560382 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.560349 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-token\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.560491 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.560453 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-envoy\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.560555 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.560513 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-data\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.560613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.560554 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-credential-socket\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.648950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.648887 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:15.661771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.661741 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-podinfo\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.661771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.661773 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-token\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.662023 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.661866 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-envoy\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.662023 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.661903 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-data\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.662023 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.661933 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-credential-socket\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.662023 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.662022 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-workload-socket\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.662362 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.662095 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5fqt\" (UniqueName: \"kubernetes.io/projected/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-kube-api-access-k5fqt\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.662362 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.662132 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-workload-certs\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.662362 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.662219 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istiod-ca-cert\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.662593 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.662567 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-workload-socket\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.662714 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.662656 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-data\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.662907 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.662881 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-credential-socket\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.663001 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.662975 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-workload-certs\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.665437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.665411 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-envoy\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.665845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.665823 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istiod-ca-cert\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.665995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.665974 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-podinfo\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.720677 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.720603 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-token\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.720677 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.720633 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5fqt\" (UniqueName: \"kubernetes.io/projected/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-kube-api-access-k5fqt\") pod \"e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.785051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.785019 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:15.877288 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.876744 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a53826-3ab4-4bc6-a88e-dcb48addd13b" path="/var/lib/kubelet/pods/61a53826-3ab4-4bc6-a88e-dcb48addd13b/volumes" Nov 02 20:44:15.926021 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:15.925973 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:16.224272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:16.224245 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc"] Nov 02 20:44:16.224635 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:44:16.224610 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb089fd4e_b4a4_4420_b7f5_f8f79b78bba5.slice/crio-92142f6e8f66a1213c4a32bd57d2293b83572af456ea9a99b33a415120e81746 WatchSource:0}: Error finding container 92142f6e8f66a1213c4a32bd57d2293b83572af456ea9a99b33a415120e81746: Status 404 returned error can't find the container with id 92142f6e8f66a1213c4a32bd57d2293b83572af456ea9a99b33a415120e81746 Nov 02 20:44:16.227797 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:16.227749 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:44:16.227918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:16.227872 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:44:16.227989 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:16.227917 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:44:16.327156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:16.327066 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" event={"ID":"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5","Type":"ContainerStarted","Data":"eb1298232bf22e960593cd92a2b66778bf1f24c8f598b3d7eb019b1b644261dd"} Nov 02 20:44:16.327329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:16.327175 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" event={"ID":"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5","Type":"ContainerStarted","Data":"92142f6e8f66a1213c4a32bd57d2293b83572af456ea9a99b33a415120e81746"} Nov 02 20:44:16.442720 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:16.442667 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" podStartSLOduration=1.442650368 podStartE2EDuration="1.442650368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:44:16.440297061 +0000 UTC m=+1784.729953590" watchObservedRunningTime="2026-11-02 20:44:16.442650368 +0000 UTC m=+1784.732306895" Nov 02 20:44:16.785520 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:16.785493 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:17.791491 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:17.791461 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:18.345503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:18.345468 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:18.346364 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:18.346340 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:44:19.510213 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:19.510147 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:19.752006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:19.751938 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:22.227202 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:22.227163 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:25.639888 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:25.639850 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:28.080135 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:28.080098 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:28.324792 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:28.324760 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:28.568185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:28.568149 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:29.401349 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:29.401317 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:30.635640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:30.635613 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2"] Nov 02 20:44:30.639965 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:30.639947 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2" Nov 02 20:44:30.694802 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:30.694773 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2"] Nov 02 20:44:30.735557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:30.735525 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:44:30.735731 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:30.735656 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:44:30.767949 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:30.767921 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2563c4e-0fe5-453d-b156-d7e11d9025b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2\" (UID: \"c2563c4e-0fe5-453d-b156-d7e11d9025b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2" Nov 02 20:44:30.768070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:30.767960 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh8rk\" (UniqueName: \"kubernetes.io/projected/c2563c4e-0fe5-453d-b156-d7e11d9025b2-kube-api-access-rh8rk\") pod \"maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2\" (UID: \"c2563c4e-0fe5-453d-b156-d7e11d9025b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2" Nov 02 20:44:30.768189 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:30.768170 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2563c4e-0fe5-453d-b156-d7e11d9025b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2\" (UID: \"c2563c4e-0fe5-453d-b156-d7e11d9025b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2" Nov 02 20:44:30.869705 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:30.869679 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2563c4e-0fe5-453d-b156-d7e11d9025b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2\" (UID: \"c2563c4e-0fe5-453d-b156-d7e11d9025b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2" Nov 02 20:44:30.869847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:30.869767 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2563c4e-0fe5-453d-b156-d7e11d9025b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2\" (UID: \"c2563c4e-0fe5-453d-b156-d7e11d9025b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2" Nov 02 20:44:30.869847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:30.869785 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh8rk\" (UniqueName: \"kubernetes.io/projected/c2563c4e-0fe5-453d-b156-d7e11d9025b2-kube-api-access-rh8rk\") pod \"maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2\" (UID: \"c2563c4e-0fe5-453d-b156-d7e11d9025b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2" Nov 02 20:44:30.870551 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:30.870530 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2563c4e-0fe5-453d-b156-d7e11d9025b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2\" (UID: \"c2563c4e-0fe5-453d-b156-d7e11d9025b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2" Nov 02 20:44:30.872278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:30.872258 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2563c4e-0fe5-453d-b156-d7e11d9025b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2\" (UID: \"c2563c4e-0fe5-453d-b156-d7e11d9025b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2" Nov 02 20:44:30.927754 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:30.927699 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh8rk\" (UniqueName: \"kubernetes.io/projected/c2563c4e-0fe5-453d-b156-d7e11d9025b2-kube-api-access-rh8rk\") pod \"maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2\" (UID: \"c2563c4e-0fe5-453d-b156-d7e11d9025b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2" Nov 02 20:44:30.929501 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:30.929476 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:30.954524 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:30.954498 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2" Nov 02 20:44:31.328280 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:44:31.328241 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2563c4e_0fe5_453d_b156_d7e11d9025b2.slice/crio-3285dfd0526908bd6ea977c559c224dc33c26d3d2c14aa1665c62e9c086f686c WatchSource:0}: Error finding container 3285dfd0526908bd6ea977c559c224dc33c26d3d2c14aa1665c62e9c086f686c: Status 404 returned error can't find the container with id 3285dfd0526908bd6ea977c559c224dc33c26d3d2c14aa1665c62e9c086f686c Nov 02 20:44:31.329379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:31.329219 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2"] Nov 02 20:44:31.503388 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:31.503358 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2" event={"ID":"c2563c4e-0fe5-453d-b156-d7e11d9025b2","Type":"ContainerStarted","Data":"3285dfd0526908bd6ea977c559c224dc33c26d3d2c14aa1665c62e9c086f686c"} Nov 02 20:44:32.082367 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:32.082292 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:32.096610 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:32.096583 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt"] Nov 02 20:44:32.100056 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:32.100037 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f1be682-gqrrt"] Nov 02 20:44:32.163015 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:32.162991 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:44:32.164936 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:32.164912 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:44:32.512812 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:32.512775 4971 generic.go:356] "Generic (PLEG): container finished" podID="c2563c4e-0fe5-453d-b156-d7e11d9025b2" containerID="1c9fc42763cbb7cd9f24bc3b3899e3e5fedf0e90e0dd05f52385e7417d39ebdf" exitCode=0 Nov 02 20:44:32.512980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:32.512865 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2" event={"ID":"c2563c4e-0fe5-453d-b156-d7e11d9025b2","Type":"ContainerDied","Data":"1c9fc42763cbb7cd9f24bc3b3899e3e5fedf0e90e0dd05f52385e7417d39ebdf"} Nov 02 20:44:33.343225 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:44:33.343188 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:44:33.402437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:33.402409 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:33.658140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:33.657926 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:33.915184 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:33.913098 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d023107-376c-4654-985b-433f2660fff2" path="/var/lib/kubelet/pods/2d023107-376c-4654-985b-433f2660fff2/volumes" Nov 02 20:44:33.920138 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:33.917121 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2" Nov 02 20:44:33.938161 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:33.938126 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2563c4e-0fe5-453d-b156-d7e11d9025b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2563c4e-0fe5-453d-b156-d7e11d9025b2-maas-api-xks-ca\") pod \"c2563c4e-0fe5-453d-b156-d7e11d9025b2\" (UID: \"c2563c4e-0fe5-453d-b156-d7e11d9025b2\") " Nov 02 20:44:33.938335 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:33.938301 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2563c4e-0fe5-453d-b156-d7e11d9025b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2563c4e-0fe5-453d-b156-d7e11d9025b2-maas-api-openshift-ca\") pod \"c2563c4e-0fe5-453d-b156-d7e11d9025b2\" (UID: \"c2563c4e-0fe5-453d-b156-d7e11d9025b2\") " Nov 02 20:44:33.938394 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:33.938372 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2563c4e-0fe5-453d-b156-d7e11d9025b2-kube-api-access-rh8rk\" (UniqueName: \"kubernetes.io/projected/c2563c4e-0fe5-453d-b156-d7e11d9025b2-kube-api-access-rh8rk\") pod \"c2563c4e-0fe5-453d-b156-d7e11d9025b2\" (UID: \"c2563c4e-0fe5-453d-b156-d7e11d9025b2\") " Nov 02 20:44:33.938715 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:33.938680 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2563c4e-0fe5-453d-b156-d7e11d9025b2-maas-api-openshift-ca" pod "c2563c4e-0fe5-453d-b156-d7e11d9025b2" (UID: "c2563c4e-0fe5-453d-b156-d7e11d9025b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:44:33.939628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:33.939140 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2563c4e-0fe5-453d-b156-d7e11d9025b2-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:33.941414 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:33.941386 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2563c4e-0fe5-453d-b156-d7e11d9025b2-maas-api-xks-ca" pod "c2563c4e-0fe5-453d-b156-d7e11d9025b2" (UID: "c2563c4e-0fe5-453d-b156-d7e11d9025b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:33.942987 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:33.941679 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2563c4e-0fe5-453d-b156-d7e11d9025b2-kube-api-access-rh8rk" pod "c2563c4e-0fe5-453d-b156-d7e11d9025b2" (UID: "c2563c4e-0fe5-453d-b156-d7e11d9025b2"). InnerVolumeSpecName "kube-api-access-rh8rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:34.040363 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:34.040331 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh8rk\" (UniqueName: \"kubernetes.io/projected/c2563c4e-0fe5-453d-b156-d7e11d9025b2-kube-api-access-rh8rk\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:34.040499 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:34.040367 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2563c4e-0fe5-453d-b156-d7e11d9025b2-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:34.566797 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:34.566758 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2" event={"ID":"c2563c4e-0fe5-453d-b156-d7e11d9025b2","Type":"ContainerDied","Data":"3285dfd0526908bd6ea977c559c224dc33c26d3d2c14aa1665c62e9c086f686c"} Nov 02 20:44:34.567237 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:34.566802 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2" Nov 02 20:44:34.567237 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:34.566818 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3285dfd0526908bd6ea977c559c224dc33c26d3d2c14aa1665c62e9c086f686c" Nov 02 20:44:35.211871 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:35.211840 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:35.455764 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:35.455731 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:40.349114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:40.348620 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:41.671167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:41.671136 4971 scope.go:122] "RemoveContainer" containerID="4bbce4920d6104364166262997e1810dd6ebbd21f65cd07e0025560a805dacfb" Nov 02 20:44:41.681815 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:41.681794 4971 scope.go:122] "RemoveContainer" containerID="5e02678173f5494faaceb6ebaa8d2e86c3b19ac58506f2267d84d36b35e0d5f2" Nov 02 20:44:41.691778 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:41.691753 4971 scope.go:122] "RemoveContainer" containerID="9e6ceb07170bbea48d266658a2d690c5ce86097fcfd51c3d04fdcb0adb1c3ca9" Nov 02 20:44:41.701249 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:41.701227 4971 scope.go:122] "RemoveContainer" containerID="170e1ecd6dd043d2228a3b0e16d730007122a58a714be11d0c276d0041f1153c" Nov 02 20:44:41.710412 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:41.710395 4971 scope.go:122] "RemoveContainer" containerID="18dcb6c6b33845851164122a39c2e41267c0c23c6df8e4a260ad1fd6bd0e15b5" Nov 02 20:44:42.286795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:42.286761 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42"] Nov 02 20:44:42.413654 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:42.413608 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp"] Nov 02 20:44:42.765148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:42.763948 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:43.008353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:43.008321 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:44.602108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:44.601480 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:46.280748 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:46.280700 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:46.534909 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:46.533683 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:46.702139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:46.702103 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l"] Nov 02 20:44:46.746853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:46.746824 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-694cee-js82l"] Nov 02 20:44:46.790275 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:46.790195 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:47.214393 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:47.212467 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:47.287488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:47.287435 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" podUID="96107d44-a6a7-4af1-840b-706c09bf59a1" containerName="payload-processing" containerID="cri-o://ee735e00b0479fb65c40adfe1f0b9778dcebaeb39ba74337739d6e1288e4dfe7" gracePeriod=25 Nov 02 20:44:47.414485 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:47.414377 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" podUID="c4e1c40c-fa5a-4707-8d39-24e1d57b0d91" containerName="payload-pre-processing" containerID="cri-o://2ba174f8e1d3678c5a27e4fc7a531d5ecfc0d151e194eecd4a69d18c95e21b19" gracePeriod=25 Nov 02 20:44:47.702218 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:47.702186 4971 generic.go:356] "Generic (PLEG): container finished" podID="fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" containerID="5cd41de231662603c3d546f5a38b51de3af269c7b4d46c17c254ac92116ddf07" exitCode=0 Nov 02 20:44:47.702218 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:47.702220 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" event={"ID":"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284","Type":"ContainerDied","Data":"5cd41de231662603c3d546f5a38b51de3af269c7b4d46c17c254ac92116ddf07"} Nov 02 20:44:47.702583 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:47.702568 4971 scope.go:122] "RemoveContainer" containerID="5cd41de231662603c3d546f5a38b51de3af269c7b4d46c17c254ac92116ddf07" Nov 02 20:44:47.703994 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:47.703961 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:44:47.704056 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:47.704036 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:44:47.704117 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:47.704070 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:44:47.870549 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:47.870511 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e672784a-3207-4ded-858f-07e0daf07b28" path="/var/lib/kubelet/pods/e672784a-3207-4ded-858f-07e0daf07b28/volumes" Nov 02 20:44:48.711445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:48.711408 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" event={"ID":"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284","Type":"ContainerStarted","Data":"e8241a558f111b3357ad5a7ef276dc4ac2884ab7b1669596ce083130d7640c0c"} Nov 02 20:44:49.646108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:49.636367 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:44:49.647220 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:49.647188 4971 patch_prober.go:29] interesting pod/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 02 20:44:49.647340 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:49.647245 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" podUID="fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 02 20:44:49.690858 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:49.690831 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" podUID="96107d44-a6a7-4af1-840b-706c09bf59a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:44:49.691517 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:49.690866 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" podUID="c4e1c40c-fa5a-4707-8d39-24e1d57b0d91" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:44:50.601995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:50.601956 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:44:50.637108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:50.633451 4971 patch_prober.go:29] interesting pod/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 02 20:44:50.647233 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:50.647191 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" podUID="fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 02 20:44:51.648735 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:51.648698 4971 patch_prober.go:29] interesting pod/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 02 20:44:51.650158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:51.649836 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" podUID="fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 02 20:44:51.805870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:51.805298 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:52.057605 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:52.057514 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:52.297783 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:52.297748 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:52.639109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:52.630832 4971 patch_prober.go:29] interesting pod/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 02 20:44:52.639109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:52.630893 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" podUID="fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 02 20:44:53.626608 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:53.626573 4971 patch_prober.go:29] interesting pod/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 02 20:44:53.627275 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:53.627229 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" podUID="fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 02 20:44:53.745118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:53.735657 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:53.983139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:53.983002 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:54.112412 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:54.112376 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6"] Nov 02 20:44:54.112690 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:54.112624 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" podUID="d0ed161a-7533-42b6-94c9-bfaacc784178" containerName="istio-proxy" containerID="cri-o://68272fa1c56e7e84119fadcb276bd18068ba62d38592cc42ea4fac96a4f0c0d0" gracePeriod=30 Nov 02 20:44:54.176260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:54.176223 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh"] Nov 02 20:44:54.176505 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:54.176470 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" podUID="ea994d9c-d955-4efa-9eea-9225a6fee64a" containerName="istio-proxy" containerID="cri-o://46935c5f2a2160f0bb05face02646a3f8d5e21a3337f943d2a32e46a7c917e0d" gracePeriod=30 Nov 02 20:44:54.181332 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:54.181306 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7"] Nov 02 20:44:54.181536 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:54.181515 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" podUID="0e0eeb03-8f43-4579-bdb9-66e80f676abb" containerName="istio-proxy" containerID="cri-o://fa66e63489b4c9d3d31e5c98f10c91e5c87822e3722d84f6f68bc5e829c0af40" gracePeriod=30 Nov 02 20:44:54.274251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:54.260316 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:54.630147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:54.630115 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:44:54.631180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:54.631158 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:44:55.083562 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.083479 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:55.687721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.687688 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn"] Nov 02 20:44:55.688376 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.688359 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c2563c4e-0fe5-453d-b156-d7e11d9025b2" containerName="revoke-keys" Nov 02 20:44:55.688376 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.688376 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2563c4e-0fe5-453d-b156-d7e11d9025b2" containerName="revoke-keys" Nov 02 20:44:55.688510 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.688501 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2563c4e-0fe5-453d-b156-d7e11d9025b2" containerName="revoke-keys" Nov 02 20:44:55.692354 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.692337 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" Nov 02 20:44:55.726733 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.726708 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn\" (UID: \"64210fb9-2c46-4cbc-b8fe-b17cf04ce960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" Nov 02 20:44:55.726869 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.726757 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzgvq\" (UniqueName: \"kubernetes.io/projected/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-kube-api-access-qzgvq\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn\" (UID: \"64210fb9-2c46-4cbc-b8fe-b17cf04ce960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" Nov 02 20:44:55.726934 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.726900 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn\" (UID: \"64210fb9-2c46-4cbc-b8fe-b17cf04ce960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" Nov 02 20:44:55.746200 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.746167 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn"] Nov 02 20:44:55.787833 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.787799 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:44:55.787959 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.787939 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:44:55.828790 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.828766 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn\" (UID: \"64210fb9-2c46-4cbc-b8fe-b17cf04ce960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" Nov 02 20:44:55.828931 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.828821 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzgvq\" (UniqueName: \"kubernetes.io/projected/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-kube-api-access-qzgvq\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn\" (UID: \"64210fb9-2c46-4cbc-b8fe-b17cf04ce960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" Nov 02 20:44:55.828931 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.828882 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn\" (UID: \"64210fb9-2c46-4cbc-b8fe-b17cf04ce960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" Nov 02 20:44:55.829445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.829411 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn\" (UID: \"64210fb9-2c46-4cbc-b8fe-b17cf04ce960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" Nov 02 20:44:55.831484 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.831460 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn\" (UID: \"64210fb9-2c46-4cbc-b8fe-b17cf04ce960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" Nov 02 20:44:55.886450 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:55.886426 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzgvq\" (UniqueName: \"kubernetes.io/projected/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-kube-api-access-qzgvq\") pod \"maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn\" (UID: \"64210fb9-2c46-4cbc-b8fe-b17cf04ce960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" Nov 02 20:44:56.006253 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:56.006196 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" Nov 02 20:44:56.383563 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:56.383534 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn"] Nov 02 20:44:56.384004 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:44:56.383975 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64210fb9_2c46_4cbc_b8fe_b17cf04ce960.slice/crio-6eac388a6db4f83ca72d97b6407b82537ba7a9d52a4fd66565015581456f733e WatchSource:0}: Error finding container 6eac388a6db4f83ca72d97b6407b82537ba7a9d52a4fd66565015581456f733e: Status 404 returned error can't find the container with id 6eac388a6db4f83ca72d97b6407b82537ba7a9d52a4fd66565015581456f733e Nov 02 20:44:57.046903 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:57.046817 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" event={"ID":"64210fb9-2c46-4cbc-b8fe-b17cf04ce960","Type":"ContainerStarted","Data":"9ff17db6e47f245e9bf57e5de71e0c370e22f9b4d24d77ed983814016076b1fa"} Nov 02 20:44:57.046903 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:57.046869 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" event={"ID":"64210fb9-2c46-4cbc-b8fe-b17cf04ce960","Type":"ContainerStarted","Data":"6eac388a6db4f83ca72d97b6407b82537ba7a9d52a4fd66565015581456f733e"} Nov 02 20:44:57.155639 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:57.155583 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" podStartSLOduration=2.155564235 podStartE2EDuration="2.155564235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:44:57.155265237 +0000 UTC m=+1825.444921765" watchObservedRunningTime="2026-11-02 20:44:57.155564235 +0000 UTC m=+1825.445220767" Nov 02 20:44:57.411715 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:57.411675 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:44:58.057017 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:58.056985 4971 generic.go:356] "Generic (PLEG): container finished" podID="64210fb9-2c46-4cbc-b8fe-b17cf04ce960" containerID="9ff17db6e47f245e9bf57e5de71e0c370e22f9b4d24d77ed983814016076b1fa" exitCode=0 Nov 02 20:44:58.057481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:58.057026 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" event={"ID":"64210fb9-2c46-4cbc-b8fe-b17cf04ce960","Type":"ContainerDied","Data":"9ff17db6e47f245e9bf57e5de71e0c370e22f9b4d24d77ed983814016076b1fa"} Nov 02 20:44:59.353640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.353612 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" Nov 02 20:44:59.478247 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.478094 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-maas-api-xks-ca\") pod \"64210fb9-2c46-4cbc-b8fe-b17cf04ce960\" (UID: \"64210fb9-2c46-4cbc-b8fe-b17cf04ce960\") " Nov 02 20:44:59.478247 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.478222 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-maas-api-openshift-ca\") pod \"64210fb9-2c46-4cbc-b8fe-b17cf04ce960\" (UID: \"64210fb9-2c46-4cbc-b8fe-b17cf04ce960\") " Nov 02 20:44:59.479755 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.478730 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-kube-api-access-qzgvq\" (UniqueName: \"kubernetes.io/projected/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-kube-api-access-qzgvq\") pod \"64210fb9-2c46-4cbc-b8fe-b17cf04ce960\" (UID: \"64210fb9-2c46-4cbc-b8fe-b17cf04ce960\") " Nov 02 20:44:59.479755 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.479032 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-maas-api-openshift-ca" pod "64210fb9-2c46-4cbc-b8fe-b17cf04ce960" (UID: "64210fb9-2c46-4cbc-b8fe-b17cf04ce960"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:44:59.479755 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.479433 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.492872 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.491515 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-maas-api-xks-ca" pod "64210fb9-2c46-4cbc-b8fe-b17cf04ce960" (UID: "64210fb9-2c46-4cbc-b8fe-b17cf04ce960"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:59.495642 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.495587 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-kube-api-access-qzgvq" pod "64210fb9-2c46-4cbc-b8fe-b17cf04ce960" (UID: "64210fb9-2c46-4cbc-b8fe-b17cf04ce960"). InnerVolumeSpecName "kube-api-access-qzgvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:59.515546 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.515518 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:44:59.541835 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.541808 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:44:59.550242 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.550216 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:44:59.580131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.580072 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-workload-socket\") pod \"d0ed161a-7533-42b6-94c9-bfaacc784178\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " Nov 02 20:44:59.580260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.580143 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-data\") pod \"d0ed161a-7533-42b6-94c9-bfaacc784178\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " Nov 02 20:44:59.580260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.580176 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-workload-certs\") pod \"d0ed161a-7533-42b6-94c9-bfaacc784178\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " Nov 02 20:44:59.580260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.580205 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-token\" (UniqueName: \"kubernetes.io/projected/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-token\") pod \"d0ed161a-7533-42b6-94c9-bfaacc784178\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " Nov 02 20:44:59.580410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.580258 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-credential-socket\") pod \"d0ed161a-7533-42b6-94c9-bfaacc784178\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " Nov 02 20:44:59.580410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.580293 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-envoy\") pod \"d0ed161a-7533-42b6-94c9-bfaacc784178\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " Nov 02 20:44:59.580410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.580291 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-workload-socket" pod "d0ed161a-7533-42b6-94c9-bfaacc784178" (UID: "d0ed161a-7533-42b6-94c9-bfaacc784178"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:59.580410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.580358 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ed161a-7533-42b6-94c9-bfaacc784178-kube-api-access-b6vmw\" (UniqueName: \"kubernetes.io/projected/d0ed161a-7533-42b6-94c9-bfaacc784178-kube-api-access-b6vmw\") pod \"d0ed161a-7533-42b6-94c9-bfaacc784178\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " Nov 02 20:44:59.580610 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.580421 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-data" pod "d0ed161a-7533-42b6-94c9-bfaacc784178" (UID: "d0ed161a-7533-42b6-94c9-bfaacc784178"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:59.580610 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.580485 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-workload-certs" pod "d0ed161a-7533-42b6-94c9-bfaacc784178" (UID: "d0ed161a-7533-42b6-94c9-bfaacc784178"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:59.580610 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.580571 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-podinfo\") pod \"d0ed161a-7533-42b6-94c9-bfaacc784178\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " Nov 02 20:44:59.580771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.580721 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0ed161a-7533-42b6-94c9-bfaacc784178-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ed161a-7533-42b6-94c9-bfaacc784178-istiod-ca-cert\") pod \"d0ed161a-7533-42b6-94c9-bfaacc784178\" (UID: \"d0ed161a-7533-42b6-94c9-bfaacc784178\") " Nov 02 20:44:59.580830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.580796 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-credential-socket" pod "d0ed161a-7533-42b6-94c9-bfaacc784178" (UID: "d0ed161a-7533-42b6-94c9-bfaacc784178"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:59.581066 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.581040 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ed161a-7533-42b6-94c9-bfaacc784178-istiod-ca-cert" pod "d0ed161a-7533-42b6-94c9-bfaacc784178" (UID: "d0ed161a-7533-42b6-94c9-bfaacc784178"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:44:59.581627 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.581604 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ed161a-7533-42b6-94c9-bfaacc784178-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.581692 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.581636 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.581692 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.581650 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.581692 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.581666 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzgvq\" (UniqueName: \"kubernetes.io/projected/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-kube-api-access-qzgvq\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.581692 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.581682 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.581828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.581697 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.581828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.581710 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64210fb9-2c46-4cbc-b8fe-b17cf04ce960-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.582628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.582607 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-envoy" pod "d0ed161a-7533-42b6-94c9-bfaacc784178" (UID: "d0ed161a-7533-42b6-94c9-bfaacc784178"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:59.582896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.582860 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-podinfo" pod "d0ed161a-7533-42b6-94c9-bfaacc784178" (UID: "d0ed161a-7533-42b6-94c9-bfaacc784178"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:44:59.583205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.583186 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-token" pod "d0ed161a-7533-42b6-94c9-bfaacc784178" (UID: "d0ed161a-7533-42b6-94c9-bfaacc784178"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:59.583258 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.583218 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ed161a-7533-42b6-94c9-bfaacc784178-kube-api-access-b6vmw" pod "d0ed161a-7533-42b6-94c9-bfaacc784178" (UID: "d0ed161a-7533-42b6-94c9-bfaacc784178"). InnerVolumeSpecName "kube-api-access-b6vmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:59.683111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.683012 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-data\") pod \"ea994d9c-d955-4efa-9eea-9225a6fee64a\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " Nov 02 20:44:59.683278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.683110 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-podinfo\") pod \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " Nov 02 20:44:59.683278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.683140 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-envoy\") pod \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " Nov 02 20:44:59.683278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.683174 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-podinfo\") pod \"ea994d9c-d955-4efa-9eea-9225a6fee64a\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " Nov 02 20:44:59.683458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.683281 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-envoy\") pod \"ea994d9c-d955-4efa-9eea-9225a6fee64a\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " Nov 02 20:44:59.683458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.683310 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-token\" (UniqueName: \"kubernetes.io/projected/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-token\") pod \"ea994d9c-d955-4efa-9eea-9225a6fee64a\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " Nov 02 20:44:59.683458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.683361 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-data" pod "ea994d9c-d955-4efa-9eea-9225a6fee64a" (UID: "ea994d9c-d955-4efa-9eea-9225a6fee64a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:59.683458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.683383 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-workload-socket\") pod \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " Nov 02 20:44:59.683458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.683431 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0eeb03-8f43-4579-bdb9-66e80f676abb-kube-api-access-94xb4\" (UniqueName: \"kubernetes.io/projected/0e0eeb03-8f43-4579-bdb9-66e80f676abb-kube-api-access-94xb4\") pod \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " Nov 02 20:44:59.683728 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.683524 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-token\" (UniqueName: \"kubernetes.io/projected/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-token\") pod \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " Nov 02 20:44:59.684405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.684119 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-workload-socket" pod "0e0eeb03-8f43-4579-bdb9-66e80f676abb" (UID: "0e0eeb03-8f43-4579-bdb9-66e80f676abb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.684883 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-credential-socket\") pod \"ea994d9c-d955-4efa-9eea-9225a6fee64a\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.684944 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-workload-certs\") pod \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.684996 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istiod-ca-cert\") pod \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.685062 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-workload-certs\") pod \"ea994d9c-d955-4efa-9eea-9225a6fee64a\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.685155 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea994d9c-d955-4efa-9eea-9225a6fee64a-kube-api-access-fthpx\" (UniqueName: \"kubernetes.io/projected/ea994d9c-d955-4efa-9eea-9225a6fee64a-kube-api-access-fthpx\") pod \"ea994d9c-d955-4efa-9eea-9225a6fee64a\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.685208 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-credential-socket\") pod \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.685315 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-data\") pod \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\" (UID: \"0e0eeb03-8f43-4579-bdb9-66e80f676abb\") " Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.685344 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-workload-socket\") pod \"ea994d9c-d955-4efa-9eea-9225a6fee64a\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.685362 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-workload-certs" pod "0e0eeb03-8f43-4579-bdb9-66e80f676abb" (UID: "0e0eeb03-8f43-4579-bdb9-66e80f676abb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.685386 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea994d9c-d955-4efa-9eea-9225a6fee64a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea994d9c-d955-4efa-9eea-9225a6fee64a-istiod-ca-cert\") pod \"ea994d9c-d955-4efa-9eea-9225a6fee64a\" (UID: \"ea994d9c-d955-4efa-9eea-9225a6fee64a\") " Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.685543 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-credential-socket" pod "ea994d9c-d955-4efa-9eea-9225a6fee64a" (UID: "ea994d9c-d955-4efa-9eea-9225a6fee64a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.686058 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.686118 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.686135 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.686141 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-data" pod "0e0eeb03-8f43-4579-bdb9-66e80f676abb" (UID: "0e0eeb03-8f43-4579-bdb9-66e80f676abb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.686150 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.686197 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6vmw\" (UniqueName: \"kubernetes.io/projected/d0ed161a-7533-42b6-94c9-bfaacc784178-kube-api-access-b6vmw\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.686202 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-envoy" pod "ea994d9c-d955-4efa-9eea-9225a6fee64a" (UID: "ea994d9c-d955-4efa-9eea-9225a6fee64a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.686218 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.686235 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ed161a-7533-42b6-94c9-bfaacc784178-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.686357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.686249 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.687646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.686402 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-workload-socket" pod "ea994d9c-d955-4efa-9eea-9225a6fee64a" (UID: "ea994d9c-d955-4efa-9eea-9225a6fee64a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:59.687646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.686462 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-credential-socket" pod "0e0eeb03-8f43-4579-bdb9-66e80f676abb" (UID: "0e0eeb03-8f43-4579-bdb9-66e80f676abb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:59.687646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.686516 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-token" pod "0e0eeb03-8f43-4579-bdb9-66e80f676abb" (UID: "0e0eeb03-8f43-4579-bdb9-66e80f676abb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:59.687646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.686661 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-workload-certs" pod "ea994d9c-d955-4efa-9eea-9225a6fee64a" (UID: "ea994d9c-d955-4efa-9eea-9225a6fee64a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:59.687646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.686699 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istiod-ca-cert" pod "0e0eeb03-8f43-4579-bdb9-66e80f676abb" (UID: "0e0eeb03-8f43-4579-bdb9-66e80f676abb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:44:59.687978 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.687950 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-podinfo" pod "ea994d9c-d955-4efa-9eea-9225a6fee64a" (UID: "ea994d9c-d955-4efa-9eea-9225a6fee64a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:44:59.688183 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.688141 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea994d9c-d955-4efa-9eea-9225a6fee64a-istiod-ca-cert" pod "ea994d9c-d955-4efa-9eea-9225a6fee64a" (UID: "ea994d9c-d955-4efa-9eea-9225a6fee64a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:44:59.690297 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.690272 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" podUID="96107d44-a6a7-4af1-840b-706c09bf59a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:44:59.690809 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.690788 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" podUID="c4e1c40c-fa5a-4707-8d39-24e1d57b0d91" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:44:59.691451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.691431 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0eeb03-8f43-4579-bdb9-66e80f676abb-kube-api-access-94xb4" pod "0e0eeb03-8f43-4579-bdb9-66e80f676abb" (UID: "0e0eeb03-8f43-4579-bdb9-66e80f676abb"). InnerVolumeSpecName "kube-api-access-94xb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:59.691451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.691441 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-envoy" pod "0e0eeb03-8f43-4579-bdb9-66e80f676abb" (UID: "0e0eeb03-8f43-4579-bdb9-66e80f676abb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:59.691590 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.691555 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-podinfo" pod "0e0eeb03-8f43-4579-bdb9-66e80f676abb" (UID: "0e0eeb03-8f43-4579-bdb9-66e80f676abb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:44:59.692038 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.692013 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-token" pod "ea994d9c-d955-4efa-9eea-9225a6fee64a" (UID: "ea994d9c-d955-4efa-9eea-9225a6fee64a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:59.692990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.692968 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea994d9c-d955-4efa-9eea-9225a6fee64a-kube-api-access-fthpx" pod "ea994d9c-d955-4efa-9eea-9225a6fee64a" (UID: "ea994d9c-d955-4efa-9eea-9225a6fee64a"). InnerVolumeSpecName "kube-api-access-fthpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:59.788251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.788215 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.788251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.788254 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fthpx\" (UniqueName: \"kubernetes.io/projected/ea994d9c-d955-4efa-9eea-9225a6fee64a-kube-api-access-fthpx\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.788447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.788271 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.788447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.788284 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.788447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.788300 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.788447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.788314 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea994d9c-d955-4efa-9eea-9225a6fee64a-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.788447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.788328 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.788447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.788341 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.788447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.788354 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.788447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.788367 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.788447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.788381 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea994d9c-d955-4efa-9eea-9225a6fee64a-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.788447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.788395 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94xb4\" (UniqueName: \"kubernetes.io/projected/0e0eeb03-8f43-4579-bdb9-66e80f676abb-kube-api-access-94xb4\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.788447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.788408 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.788447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.788421 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0eeb03-8f43-4579-bdb9-66e80f676abb-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.900600 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:44:59.899136 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:00.073413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.073337 4971 generic.go:356] "Generic (PLEG): container finished" podID="d0ed161a-7533-42b6-94c9-bfaacc784178" containerID="68272fa1c56e7e84119fadcb276bd18068ba62d38592cc42ea4fac96a4f0c0d0" exitCode=0 Nov 02 20:45:00.073413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.073389 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" Nov 02 20:45:00.073615 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.073426 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" event={"ID":"d0ed161a-7533-42b6-94c9-bfaacc784178","Type":"ContainerDied","Data":"68272fa1c56e7e84119fadcb276bd18068ba62d38592cc42ea4fac96a4f0c0d0"} Nov 02 20:45:00.073615 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.073486 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6" event={"ID":"d0ed161a-7533-42b6-94c9-bfaacc784178","Type":"ContainerDied","Data":"d1e265de2b87e66f8546de88dcfe47aae40deff4b8c8ab5e721a801b089af41a"} Nov 02 20:45:00.073615 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.073509 4971 scope.go:122] "RemoveContainer" containerID="68272fa1c56e7e84119fadcb276bd18068ba62d38592cc42ea4fac96a4f0c0d0" Nov 02 20:45:00.076158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.075555 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" Nov 02 20:45:00.076158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.075607 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn" event={"ID":"64210fb9-2c46-4cbc-b8fe-b17cf04ce960","Type":"ContainerDied","Data":"6eac388a6db4f83ca72d97b6407b82537ba7a9d52a4fd66565015581456f733e"} Nov 02 20:45:00.076158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.075668 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eac388a6db4f83ca72d97b6407b82537ba7a9d52a4fd66565015581456f733e" Nov 02 20:45:00.077949 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.077925 4971 generic.go:356] "Generic (PLEG): container finished" podID="0e0eeb03-8f43-4579-bdb9-66e80f676abb" containerID="fa66e63489b4c9d3d31e5c98f10c91e5c87822e3722d84f6f68bc5e829c0af40" exitCode=0 Nov 02 20:45:00.078096 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.077980 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" Nov 02 20:45:00.078096 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.078011 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" event={"ID":"0e0eeb03-8f43-4579-bdb9-66e80f676abb","Type":"ContainerDied","Data":"fa66e63489b4c9d3d31e5c98f10c91e5c87822e3722d84f6f68bc5e829c0af40"} Nov 02 20:45:00.078096 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.078067 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7" event={"ID":"0e0eeb03-8f43-4579-bdb9-66e80f676abb","Type":"ContainerDied","Data":"ac1a002f50ab871ec7e7a91d185c429ad58763136e4e701bbde83922c907b2c1"} Nov 02 20:45:00.079698 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.079672 4971 generic.go:356] "Generic (PLEG): container finished" podID="ea994d9c-d955-4efa-9eea-9225a6fee64a" containerID="46935c5f2a2160f0bb05face02646a3f8d5e21a3337f943d2a32e46a7c917e0d" exitCode=0 Nov 02 20:45:00.079859 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.079719 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" Nov 02 20:45:00.079859 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.079738 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" event={"ID":"ea994d9c-d955-4efa-9eea-9225a6fee64a","Type":"ContainerDied","Data":"46935c5f2a2160f0bb05face02646a3f8d5e21a3337f943d2a32e46a7c917e0d"} Nov 02 20:45:00.079859 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.079777 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh" event={"ID":"ea994d9c-d955-4efa-9eea-9225a6fee64a","Type":"ContainerDied","Data":"eb54c54072cdc3df4ff15d58bbec2a4e3329aa250524b21ac56d031504de742b"} Nov 02 20:45:00.086610 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.086580 4971 scope.go:122] "RemoveContainer" containerID="68272fa1c56e7e84119fadcb276bd18068ba62d38592cc42ea4fac96a4f0c0d0" Nov 02 20:45:00.086881 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:00.086863 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68272fa1c56e7e84119fadcb276bd18068ba62d38592cc42ea4fac96a4f0c0d0\": container with ID starting with 68272fa1c56e7e84119fadcb276bd18068ba62d38592cc42ea4fac96a4f0c0d0 not found: ID does not exist" containerID="68272fa1c56e7e84119fadcb276bd18068ba62d38592cc42ea4fac96a4f0c0d0" Nov 02 20:45:00.086933 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.086888 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68272fa1c56e7e84119fadcb276bd18068ba62d38592cc42ea4fac96a4f0c0d0"} err="failed to get container status \"68272fa1c56e7e84119fadcb276bd18068ba62d38592cc42ea4fac96a4f0c0d0\": rpc error: code = NotFound desc = could not find container \"68272fa1c56e7e84119fadcb276bd18068ba62d38592cc42ea4fac96a4f0c0d0\": container with ID starting with 68272fa1c56e7e84119fadcb276bd18068ba62d38592cc42ea4fac96a4f0c0d0 not found: ID does not exist" Nov 02 20:45:00.086933 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.086904 4971 scope.go:122] "RemoveContainer" containerID="fa66e63489b4c9d3d31e5c98f10c91e5c87822e3722d84f6f68bc5e829c0af40" Nov 02 20:45:00.097352 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.097333 4971 scope.go:122] "RemoveContainer" containerID="c6bce39071dea22b90e40e0a91e3a1cd7bf1f4142c20aad00c5a8b11a40924b8" Nov 02 20:45:00.097969 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.097947 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw"] Nov 02 20:45:00.109583 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.109557 4971 scope.go:122] "RemoveContainer" containerID="fa66e63489b4c9d3d31e5c98f10c91e5c87822e3722d84f6f68bc5e829c0af40" Nov 02 20:45:00.109905 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:00.109885 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa66e63489b4c9d3d31e5c98f10c91e5c87822e3722d84f6f68bc5e829c0af40\": container with ID starting with fa66e63489b4c9d3d31e5c98f10c91e5c87822e3722d84f6f68bc5e829c0af40 not found: ID does not exist" containerID="fa66e63489b4c9d3d31e5c98f10c91e5c87822e3722d84f6f68bc5e829c0af40" Nov 02 20:45:00.109977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.109913 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa66e63489b4c9d3d31e5c98f10c91e5c87822e3722d84f6f68bc5e829c0af40"} err="failed to get container status \"fa66e63489b4c9d3d31e5c98f10c91e5c87822e3722d84f6f68bc5e829c0af40\": rpc error: code = NotFound desc = could not find container \"fa66e63489b4c9d3d31e5c98f10c91e5c87822e3722d84f6f68bc5e829c0af40\": container with ID starting with fa66e63489b4c9d3d31e5c98f10c91e5c87822e3722d84f6f68bc5e829c0af40 not found: ID does not exist" Nov 02 20:45:00.109977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.109930 4971 scope.go:122] "RemoveContainer" containerID="c6bce39071dea22b90e40e0a91e3a1cd7bf1f4142c20aad00c5a8b11a40924b8" Nov 02 20:45:00.110254 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:00.110235 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6bce39071dea22b90e40e0a91e3a1cd7bf1f4142c20aad00c5a8b11a40924b8\": container with ID starting with c6bce39071dea22b90e40e0a91e3a1cd7bf1f4142c20aad00c5a8b11a40924b8 not found: ID does not exist" containerID="c6bce39071dea22b90e40e0a91e3a1cd7bf1f4142c20aad00c5a8b11a40924b8" Nov 02 20:45:00.110330 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.110260 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6bce39071dea22b90e40e0a91e3a1cd7bf1f4142c20aad00c5a8b11a40924b8"} err="failed to get container status \"c6bce39071dea22b90e40e0a91e3a1cd7bf1f4142c20aad00c5a8b11a40924b8\": rpc error: code = NotFound desc = could not find container \"c6bce39071dea22b90e40e0a91e3a1cd7bf1f4142c20aad00c5a8b11a40924b8\": container with ID starting with c6bce39071dea22b90e40e0a91e3a1cd7bf1f4142c20aad00c5a8b11a40924b8 not found: ID does not exist" Nov 02 20:45:00.110330 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.110275 4971 scope.go:122] "RemoveContainer" containerID="46935c5f2a2160f0bb05face02646a3f8d5e21a3337f943d2a32e46a7c917e0d" Nov 02 20:45:00.117698 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.117667 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2"] Nov 02 20:45:00.122229 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.122206 4971 scope.go:122] "RemoveContainer" containerID="46935c5f2a2160f0bb05face02646a3f8d5e21a3337f943d2a32e46a7c917e0d" Nov 02 20:45:00.122501 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:00.122482 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46935c5f2a2160f0bb05face02646a3f8d5e21a3337f943d2a32e46a7c917e0d\": container with ID starting with 46935c5f2a2160f0bb05face02646a3f8d5e21a3337f943d2a32e46a7c917e0d not found: ID does not exist" containerID="46935c5f2a2160f0bb05face02646a3f8d5e21a3337f943d2a32e46a7c917e0d" Nov 02 20:45:00.122580 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.122513 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46935c5f2a2160f0bb05face02646a3f8d5e21a3337f943d2a32e46a7c917e0d"} err="failed to get container status \"46935c5f2a2160f0bb05face02646a3f8d5e21a3337f943d2a32e46a7c917e0d\": rpc error: code = NotFound desc = could not find container \"46935c5f2a2160f0bb05face02646a3f8d5e21a3337f943d2a32e46a7c917e0d\": container with ID starting with 46935c5f2a2160f0bb05face02646a3f8d5e21a3337f943d2a32e46a7c917e0d not found: ID does not exist" Nov 02 20:45:00.172910 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.172884 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr"] Nov 02 20:45:00.173837 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.173816 4971 cpu_manager.go:414] "containerMap: removing container" podUID="d0ed161a-7533-42b6-94c9-bfaacc784178" containerName="istio-proxy" Nov 02 20:45:00.173837 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.173837 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0ed161a-7533-42b6-94c9-bfaacc784178" containerName="istio-proxy" Nov 02 20:45:00.173979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.173869 4971 cpu_manager.go:414] "containerMap: removing container" podUID="0e0eeb03-8f43-4579-bdb9-66e80f676abb" containerName="istio-proxy" Nov 02 20:45:00.173979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.173874 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0eeb03-8f43-4579-bdb9-66e80f676abb" containerName="istio-proxy" Nov 02 20:45:00.173979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.173887 4971 cpu_manager.go:414] "containerMap: removing container" podUID="0e0eeb03-8f43-4579-bdb9-66e80f676abb" containerName="istio-proxy" Nov 02 20:45:00.173979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.173893 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0eeb03-8f43-4579-bdb9-66e80f676abb" containerName="istio-proxy" Nov 02 20:45:00.173979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.173907 4971 cpu_manager.go:414] "containerMap: removing container" podUID="ea994d9c-d955-4efa-9eea-9225a6fee64a" containerName="istio-proxy" Nov 02 20:45:00.173979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.173911 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea994d9c-d955-4efa-9eea-9225a6fee64a" containerName="istio-proxy" Nov 02 20:45:00.173979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.173921 4971 cpu_manager.go:414] "containerMap: removing container" podUID="64210fb9-2c46-4cbc-b8fe-b17cf04ce960" containerName="revoke-keys" Nov 02 20:45:00.173979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.173926 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64210fb9-2c46-4cbc-b8fe-b17cf04ce960" containerName="revoke-keys" Nov 02 20:45:00.174288 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.174012 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="64210fb9-2c46-4cbc-b8fe-b17cf04ce960" containerName="revoke-keys" Nov 02 20:45:00.174288 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.174022 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0ed161a-7533-42b6-94c9-bfaacc784178" containerName="istio-proxy" Nov 02 20:45:00.174288 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.174033 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0eeb03-8f43-4579-bdb9-66e80f676abb" containerName="istio-proxy" Nov 02 20:45:00.174288 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.174040 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0eeb03-8f43-4579-bdb9-66e80f676abb" containerName="istio-proxy" Nov 02 20:45:00.174288 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.174047 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea994d9c-d955-4efa-9eea-9225a6fee64a" containerName="istio-proxy" Nov 02 20:45:00.179278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.179260 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47"] Nov 02 20:45:00.179455 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.179374 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" Nov 02 20:45:00.184019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.183696 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2"] Nov 02 20:45:00.184019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.183792 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" Nov 02 20:45:00.188704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.188682 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c"] Nov 02 20:45:00.188908 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.188889 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" Nov 02 20:45:00.192801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.192784 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf"] Nov 02 20:45:00.194271 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.194251 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" Nov 02 20:45:00.201760 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.201735 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x"] Nov 02 20:45:00.201973 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.201835 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" Nov 02 20:45:00.206654 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.206635 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl"] Nov 02 20:45:00.206740 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.206729 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" Nov 02 20:45:00.211754 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.211734 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" Nov 02 20:45:00.229221 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.225559 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc"] Nov 02 20:45:00.230824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.230567 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:45:00.235493 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.235468 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" Nov 02 20:45:00.239273 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.239247 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr"] Nov 02 20:45:00.241053 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.241030 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47"] Nov 02 20:45:00.258724 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.257155 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7"] Nov 02 20:45:00.258724 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.257211 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2"] Nov 02 20:45:00.262132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.262055 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf"] Nov 02 20:45:00.265242 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.265219 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c"] Nov 02 20:45:00.271522 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.271503 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-qsqc7"] Nov 02 20:45:00.301355 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.301201 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7c43be-2b76-4027-8a05-24c8b6bedea5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47\" (UID: \"5b7c43be-2b76-4027-8a05-24c8b6bedea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" Nov 02 20:45:00.301355 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.301281 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54901b53-c9d1-43c1-a587-63845bb341cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c\" (UID: \"54901b53-c9d1-43c1-a587-63845bb341cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" Nov 02 20:45:00.301549 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.301495 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptq9q\" (UniqueName: \"kubernetes.io/projected/54901b53-c9d1-43c1-a587-63845bb341cc-kube-api-access-ptq9q\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c\" (UID: \"54901b53-c9d1-43c1-a587-63845bb341cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" Nov 02 20:45:00.301549 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.301535 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw8gw\" (UniqueName: \"kubernetes.io/projected/6f52b903-f5e5-4ade-b79e-12c5a1f79ff5-kube-api-access-gw8gw\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2\" (UID: \"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" Nov 02 20:45:00.301667 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.301585 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bf8d79-25c9-453a-8395-ca13f8f03d16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl\" (UID: \"b0bf8d79-25c9-453a-8395-ca13f8f03d16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" Nov 02 20:45:00.301667 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.301622 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jngjg\" (UniqueName: \"kubernetes.io/projected/c5cb9679-123e-4d5b-9452-82ae0fac9fe7-kube-api-access-jngjg\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr\" (UID: \"c5cb9679-123e-4d5b-9452-82ae0fac9fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" Nov 02 20:45:00.301779 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.301699 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc6xb\" (UniqueName: \"kubernetes.io/projected/b0bf8d79-25c9-453a-8395-ca13f8f03d16-kube-api-access-pc6xb\") pod \"maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl\" (UID: \"b0bf8d79-25c9-453a-8395-ca13f8f03d16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" Nov 02 20:45:00.301838 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.301788 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5cb9679-123e-4d5b-9452-82ae0fac9fe7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr\" (UID: \"c5cb9679-123e-4d5b-9452-82ae0fac9fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" Nov 02 20:45:00.301900 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.301838 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg96l\" (UniqueName: \"kubernetes.io/projected/45bc19b5-e425-4cc1-85b1-2b0f25751cfa-kube-api-access-pg96l\") pod \"maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x\" (UID: \"45bc19b5-e425-4cc1-85b1-2b0f25751cfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" Nov 02 20:45:00.301952 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.301919 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45bc19b5-e425-4cc1-85b1-2b0f25751cfa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x\" (UID: \"45bc19b5-e425-4cc1-85b1-2b0f25751cfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" Nov 02 20:45:00.302055 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.302015 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vc6l\" (UniqueName: \"kubernetes.io/projected/5b7c43be-2b76-4027-8a05-24c8b6bedea5-kube-api-access-2vc6l\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47\" (UID: \"5b7c43be-2b76-4027-8a05-24c8b6bedea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" Nov 02 20:45:00.302149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.302074 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37c9b65e-f647-4752-8b1e-ab04fb5257ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc\" (UID: \"37c9b65e-f647-4752-8b1e-ab04fb5257ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" Nov 02 20:45:00.302225 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.302188 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znb6q\" (UniqueName: \"kubernetes.io/projected/7d07f1a3-680b-46aa-9a1a-2f7c0911d411-kube-api-access-znb6q\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf\" (UID: \"7d07f1a3-680b-46aa-9a1a-2f7c0911d411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" Nov 02 20:45:00.302290 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.302248 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d07f1a3-680b-46aa-9a1a-2f7c0911d411-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf\" (UID: \"7d07f1a3-680b-46aa-9a1a-2f7c0911d411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" Nov 02 20:45:00.302342 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.302314 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f52b903-f5e5-4ade-b79e-12c5a1f79ff5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2\" (UID: \"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" Nov 02 20:45:00.302405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.302366 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cj7t\" (UniqueName: \"kubernetes.io/projected/37c9b65e-f647-4752-8b1e-ab04fb5257ec-kube-api-access-2cj7t\") pod \"maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc\" (UID: \"37c9b65e-f647-4752-8b1e-ab04fb5257ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" Nov 02 20:45:00.313064 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.310380 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl"] Nov 02 20:45:00.337496 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.337473 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc"] Nov 02 20:45:00.341755 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.341728 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x"] Nov 02 20:45:00.346406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.346382 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:00.403571 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.403551 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bf8d79-25c9-453a-8395-ca13f8f03d16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl\" (UID: \"b0bf8d79-25c9-453a-8395-ca13f8f03d16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" Nov 02 20:45:00.403867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.403586 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jngjg\" (UniqueName: \"kubernetes.io/projected/c5cb9679-123e-4d5b-9452-82ae0fac9fe7-kube-api-access-jngjg\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr\" (UID: \"c5cb9679-123e-4d5b-9452-82ae0fac9fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" Nov 02 20:45:00.403867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.403610 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6xb\" (UniqueName: \"kubernetes.io/projected/b0bf8d79-25c9-453a-8395-ca13f8f03d16-kube-api-access-pc6xb\") pod \"maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl\" (UID: \"b0bf8d79-25c9-453a-8395-ca13f8f03d16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" Nov 02 20:45:00.403867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.403734 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5cb9679-123e-4d5b-9452-82ae0fac9fe7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr\" (UID: \"c5cb9679-123e-4d5b-9452-82ae0fac9fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" Nov 02 20:45:00.403867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.403776 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg96l\" (UniqueName: \"kubernetes.io/projected/45bc19b5-e425-4cc1-85b1-2b0f25751cfa-kube-api-access-pg96l\") pod \"maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x\" (UID: \"45bc19b5-e425-4cc1-85b1-2b0f25751cfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" Nov 02 20:45:00.403867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.403833 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45bc19b5-e425-4cc1-85b1-2b0f25751cfa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x\" (UID: \"45bc19b5-e425-4cc1-85b1-2b0f25751cfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" Nov 02 20:45:00.404133 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.403873 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vc6l\" (UniqueName: \"kubernetes.io/projected/5b7c43be-2b76-4027-8a05-24c8b6bedea5-kube-api-access-2vc6l\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47\" (UID: \"5b7c43be-2b76-4027-8a05-24c8b6bedea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" Nov 02 20:45:00.404133 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.403910 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37c9b65e-f647-4752-8b1e-ab04fb5257ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc\" (UID: \"37c9b65e-f647-4752-8b1e-ab04fb5257ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" Nov 02 20:45:00.404133 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.404099 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znb6q\" (UniqueName: \"kubernetes.io/projected/7d07f1a3-680b-46aa-9a1a-2f7c0911d411-kube-api-access-znb6q\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf\" (UID: \"7d07f1a3-680b-46aa-9a1a-2f7c0911d411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" Nov 02 20:45:00.404316 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.404161 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d07f1a3-680b-46aa-9a1a-2f7c0911d411-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf\" (UID: \"7d07f1a3-680b-46aa-9a1a-2f7c0911d411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" Nov 02 20:45:00.404316 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.404235 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f52b903-f5e5-4ade-b79e-12c5a1f79ff5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2\" (UID: \"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" Nov 02 20:45:00.404316 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.404283 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cj7t\" (UniqueName: \"kubernetes.io/projected/37c9b65e-f647-4752-8b1e-ab04fb5257ec-kube-api-access-2cj7t\") pod \"maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc\" (UID: \"37c9b65e-f647-4752-8b1e-ab04fb5257ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" Nov 02 20:45:00.404490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.404332 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7c43be-2b76-4027-8a05-24c8b6bedea5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47\" (UID: \"5b7c43be-2b76-4027-8a05-24c8b6bedea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" Nov 02 20:45:00.404490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.404338 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bf8d79-25c9-453a-8395-ca13f8f03d16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl\" (UID: \"b0bf8d79-25c9-453a-8395-ca13f8f03d16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" Nov 02 20:45:00.404641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.404548 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54901b53-c9d1-43c1-a587-63845bb341cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c\" (UID: \"54901b53-c9d1-43c1-a587-63845bb341cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" Nov 02 20:45:00.404641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.404580 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45bc19b5-e425-4cc1-85b1-2b0f25751cfa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x\" (UID: \"45bc19b5-e425-4cc1-85b1-2b0f25751cfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" Nov 02 20:45:00.404641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.404589 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5cb9679-123e-4d5b-9452-82ae0fac9fe7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr\" (UID: \"c5cb9679-123e-4d5b-9452-82ae0fac9fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" Nov 02 20:45:00.404641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.404628 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37c9b65e-f647-4752-8b1e-ab04fb5257ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc\" (UID: \"37c9b65e-f647-4752-8b1e-ab04fb5257ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" Nov 02 20:45:00.404872 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.404704 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptq9q\" (UniqueName: \"kubernetes.io/projected/54901b53-c9d1-43c1-a587-63845bb341cc-kube-api-access-ptq9q\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c\" (UID: \"54901b53-c9d1-43c1-a587-63845bb341cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" Nov 02 20:45:00.404872 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.404755 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw8gw\" (UniqueName: \"kubernetes.io/projected/6f52b903-f5e5-4ade-b79e-12c5a1f79ff5-kube-api-access-gw8gw\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2\" (UID: \"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" Nov 02 20:45:00.405040 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.405017 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7c43be-2b76-4027-8a05-24c8b6bedea5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47\" (UID: \"5b7c43be-2b76-4027-8a05-24c8b6bedea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" Nov 02 20:45:00.405132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.405041 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f52b903-f5e5-4ade-b79e-12c5a1f79ff5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2\" (UID: \"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" Nov 02 20:45:00.405132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.405038 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d07f1a3-680b-46aa-9a1a-2f7c0911d411-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf\" (UID: \"7d07f1a3-680b-46aa-9a1a-2f7c0911d411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" Nov 02 20:45:00.405218 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.405201 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54901b53-c9d1-43c1-a587-63845bb341cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c\" (UID: \"54901b53-c9d1-43c1-a587-63845bb341cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" Nov 02 20:45:00.466860 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.466828 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptq9q\" (UniqueName: \"kubernetes.io/projected/54901b53-c9d1-43c1-a587-63845bb341cc-kube-api-access-ptq9q\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c\" (UID: \"54901b53-c9d1-43c1-a587-63845bb341cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" Nov 02 20:45:00.470785 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.470755 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw8gw\" (UniqueName: \"kubernetes.io/projected/6f52b903-f5e5-4ade-b79e-12c5a1f79ff5-kube-api-access-gw8gw\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2\" (UID: \"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" Nov 02 20:45:00.470930 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.470795 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vc6l\" (UniqueName: \"kubernetes.io/projected/5b7c43be-2b76-4027-8a05-24c8b6bedea5-kube-api-access-2vc6l\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47\" (UID: \"5b7c43be-2b76-4027-8a05-24c8b6bedea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" Nov 02 20:45:00.471456 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.471431 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znb6q\" (UniqueName: \"kubernetes.io/projected/7d07f1a3-680b-46aa-9a1a-2f7c0911d411-kube-api-access-znb6q\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf\" (UID: \"7d07f1a3-680b-46aa-9a1a-2f7c0911d411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" Nov 02 20:45:00.471552 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.471494 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg96l\" (UniqueName: \"kubernetes.io/projected/45bc19b5-e425-4cc1-85b1-2b0f25751cfa-kube-api-access-pg96l\") pod \"maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x\" (UID: \"45bc19b5-e425-4cc1-85b1-2b0f25751cfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" Nov 02 20:45:00.472704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.472023 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jngjg\" (UniqueName: \"kubernetes.io/projected/c5cb9679-123e-4d5b-9452-82ae0fac9fe7-kube-api-access-jngjg\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr\" (UID: \"c5cb9679-123e-4d5b-9452-82ae0fac9fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" Nov 02 20:45:00.472704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.472668 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cj7t\" (UniqueName: \"kubernetes.io/projected/37c9b65e-f647-4752-8b1e-ab04fb5257ec-kube-api-access-2cj7t\") pod \"maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc\" (UID: \"37c9b65e-f647-4752-8b1e-ab04fb5257ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" Nov 02 20:45:00.473828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.473783 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc6xb\" (UniqueName: \"kubernetes.io/projected/b0bf8d79-25c9-453a-8395-ca13f8f03d16-kube-api-access-pc6xb\") pod \"maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl\" (UID: \"b0bf8d79-25c9-453a-8395-ca13f8f03d16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" Nov 02 20:45:00.519703 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.519678 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6"] Nov 02 20:45:00.523756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.523736 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-zmzk6"] Nov 02 20:45:00.588765 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.587471 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" Nov 02 20:45:00.630424 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.630389 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:00.631251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.631224 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" Nov 02 20:45:00.640149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.640122 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" Nov 02 20:45:00.655197 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.654967 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" Nov 02 20:45:00.664636 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.664002 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" Nov 02 20:45:00.676135 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.675875 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" Nov 02 20:45:00.683257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.683227 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" Nov 02 20:45:00.689327 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.689271 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" Nov 02 20:45:00.803984 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.802567 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh"] Nov 02 20:45:00.806983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.806950 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-2q5dh"] Nov 02 20:45:00.941813 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:00.941212 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:01.204975 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:01.204865 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:01.235949 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:45:01.235922 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5cb9679_123e_4d5b_9452_82ae0fac9fe7.slice/crio-3593cb20d0645da98c97517e230f9c7591bf7b1cc28df5e6bb75a916ba9b5634 WatchSource:0}: Error finding container 3593cb20d0645da98c97517e230f9c7591bf7b1cc28df5e6bb75a916ba9b5634: Status 404 returned error can't find the container with id 3593cb20d0645da98c97517e230f9c7591bf7b1cc28df5e6bb75a916ba9b5634 Nov 02 20:45:01.237295 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:01.237272 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr"] Nov 02 20:45:01.268213 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:45:01.268189 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7c43be_2b76_4027_8a05_24c8b6bedea5.slice/crio-e1f6c9abe9523e464360459a0a56c00f784f99010b2b1fbec27dc596656695b2 WatchSource:0}: Error finding container e1f6c9abe9523e464360459a0a56c00f784f99010b2b1fbec27dc596656695b2: Status 404 returned error can't find the container with id e1f6c9abe9523e464360459a0a56c00f784f99010b2b1fbec27dc596656695b2 Nov 02 20:45:01.270434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:01.270398 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47"] Nov 02 20:45:01.281538 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:01.281489 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c"] Nov 02 20:45:01.284265 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:45:01.284237 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54901b53_c9d1_43c1_a587_63845bb341cc.slice/crio-991eb98ebae183396cacd49838da48715214fab607a3da7ce9b2b623c40ea4af WatchSource:0}: Error finding container 991eb98ebae183396cacd49838da48715214fab607a3da7ce9b2b623c40ea4af: Status 404 returned error can't find the container with id 991eb98ebae183396cacd49838da48715214fab607a3da7ce9b2b623c40ea4af Nov 02 20:45:01.285731 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:45:01.285658 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bf8d79_25c9_453a_8395_ca13f8f03d16.slice/crio-ebe337256aaba98d742c9f218bafb2ce1d4fe48db35c4d2968084cff170e9190 WatchSource:0}: Error finding container ebe337256aaba98d742c9f218bafb2ce1d4fe48db35c4d2968084cff170e9190: Status 404 returned error can't find the container with id ebe337256aaba98d742c9f218bafb2ce1d4fe48db35c4d2968084cff170e9190 Nov 02 20:45:01.288311 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:01.288285 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl"] Nov 02 20:45:01.294995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:01.294975 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf"] Nov 02 20:45:01.306242 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:45:01.305329 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d07f1a3_680b_46aa_9a1a_2f7c0911d411.slice/crio-6bd64372784920ea2ffa2f19459e948574a97883d72a7ee8ac3c47159f36da91 WatchSource:0}: Error finding container 6bd64372784920ea2ffa2f19459e948574a97883d72a7ee8ac3c47159f36da91: Status 404 returned error can't find the container with id 6bd64372784920ea2ffa2f19459e948574a97883d72a7ee8ac3c47159f36da91 Nov 02 20:45:01.363904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:01.363881 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc"] Nov 02 20:45:01.367527 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:01.367504 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x"] Nov 02 20:45:01.383945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:01.383923 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2"] Nov 02 20:45:01.540734 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:45:01.540681 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c9b65e_f647_4752_8b1e_ab04fb5257ec.slice/crio-36dd67d003ab4e784c9b296e0df8656ff5fe7e96f7bf0ba395511798886f6031 WatchSource:0}: Error finding container 36dd67d003ab4e784c9b296e0df8656ff5fe7e96f7bf0ba395511798886f6031: Status 404 returned error can't find the container with id 36dd67d003ab4e784c9b296e0df8656ff5fe7e96f7bf0ba395511798886f6031 Nov 02 20:45:01.542204 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:45:01.542159 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bc19b5_e425_4cc1_85b1_2b0f25751cfa.slice/crio-925947f2f4b95281e9c9d2548596e6b5f2cb8f26e1a5b335b6d3c5aa1fbd9e59 WatchSource:0}: Error finding container 925947f2f4b95281e9c9d2548596e6b5f2cb8f26e1a5b335b6d3c5aa1fbd9e59: Status 404 returned error can't find the container with id 925947f2f4b95281e9c9d2548596e6b5f2cb8f26e1a5b335b6d3c5aa1fbd9e59 Nov 02 20:45:01.878917 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:01.878888 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e0eeb03-8f43-4579-bdb9-66e80f676abb" path="/var/lib/kubelet/pods/0e0eeb03-8f43-4579-bdb9-66e80f676abb/volumes" Nov 02 20:45:01.879440 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:01.879423 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0ed161a-7533-42b6-94c9-bfaacc784178" path="/var/lib/kubelet/pods/d0ed161a-7533-42b6-94c9-bfaacc784178/volumes" Nov 02 20:45:01.879873 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:01.879860 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea994d9c-d955-4efa-9eea-9225a6fee64a" path="/var/lib/kubelet/pods/ea994d9c-d955-4efa-9eea-9225a6fee64a/volumes" Nov 02 20:45:02.101011 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.100974 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" event={"ID":"7d07f1a3-680b-46aa-9a1a-2f7c0911d411","Type":"ContainerStarted","Data":"6bd64372784920ea2ffa2f19459e948574a97883d72a7ee8ac3c47159f36da91"} Nov 02 20:45:02.102652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.102616 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" event={"ID":"c5cb9679-123e-4d5b-9452-82ae0fac9fe7","Type":"ContainerStarted","Data":"ee4d3f5a11a67132fb2a903be5b7aaf4859db49491e765c82d79866d595a1632"} Nov 02 20:45:02.102782 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.102667 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" event={"ID":"c5cb9679-123e-4d5b-9452-82ae0fac9fe7","Type":"ContainerStarted","Data":"3593cb20d0645da98c97517e230f9c7591bf7b1cc28df5e6bb75a916ba9b5634"} Nov 02 20:45:02.104351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.104328 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" event={"ID":"37c9b65e-f647-4752-8b1e-ab04fb5257ec","Type":"ContainerStarted","Data":"050b1f427c968350eefa8cc242b00eca1e896a7da4f7a7c0e7bfe71e16a7a642"} Nov 02 20:45:02.104467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.104370 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" event={"ID":"37c9b65e-f647-4752-8b1e-ab04fb5257ec","Type":"ContainerStarted","Data":"36dd67d003ab4e784c9b296e0df8656ff5fe7e96f7bf0ba395511798886f6031"} Nov 02 20:45:02.106240 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.106209 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" event={"ID":"b0bf8d79-25c9-453a-8395-ca13f8f03d16","Type":"ContainerStarted","Data":"67575b0e9fc743d9c52da9001e99d43c74ef82d05bedfb8073ff794a067caa79"} Nov 02 20:45:02.106331 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.106264 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" event={"ID":"b0bf8d79-25c9-453a-8395-ca13f8f03d16","Type":"ContainerStarted","Data":"ebe337256aaba98d742c9f218bafb2ce1d4fe48db35c4d2968084cff170e9190"} Nov 02 20:45:02.107873 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.107845 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" event={"ID":"54901b53-c9d1-43c1-a587-63845bb341cc","Type":"ContainerStarted","Data":"701cf425b6fe185b1f03f496011ca502cd4d08d3151c0e757a883041edb63ee8"} Nov 02 20:45:02.107983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.107889 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" event={"ID":"54901b53-c9d1-43c1-a587-63845bb341cc","Type":"ContainerStarted","Data":"991eb98ebae183396cacd49838da48715214fab607a3da7ce9b2b623c40ea4af"} Nov 02 20:45:02.109490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.109461 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" event={"ID":"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5","Type":"ContainerStarted","Data":"bbf7aad5658555589bcbedd2236536a068e20df67961ab56e9204de392d28af5"} Nov 02 20:45:02.109585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.109511 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" event={"ID":"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5","Type":"ContainerStarted","Data":"cee1f7c2c04342994f5fae6a7b40d551ec897529e7c67c45099139590655e1f5"} Nov 02 20:45:02.110906 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.110885 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" event={"ID":"5b7c43be-2b76-4027-8a05-24c8b6bedea5","Type":"ContainerStarted","Data":"79e54665e20f64a8b59d9cb1bf68175658c558de7e878853e7d78f31f7ac83a6"} Nov 02 20:45:02.111004 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.110923 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" event={"ID":"5b7c43be-2b76-4027-8a05-24c8b6bedea5","Type":"ContainerStarted","Data":"e1f6c9abe9523e464360459a0a56c00f784f99010b2b1fbec27dc596656695b2"} Nov 02 20:45:02.112213 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.112191 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" event={"ID":"45bc19b5-e425-4cc1-85b1-2b0f25751cfa","Type":"ContainerStarted","Data":"925947f2f4b95281e9c9d2548596e6b5f2cb8f26e1a5b335b6d3c5aa1fbd9e59"} Nov 02 20:45:02.215261 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.215215 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" podStartSLOduration=2.215201829 podStartE2EDuration="2.215201829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:02.211715866 +0000 UTC m=+1830.501372398" watchObservedRunningTime="2026-11-02 20:45:02.215201829 +0000 UTC m=+1830.504858357" Nov 02 20:45:02.323480 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.323425 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" podStartSLOduration=2.323407534 podStartE2EDuration="2.323407534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:02.32047647 +0000 UTC m=+1830.610133000" watchObservedRunningTime="2026-11-02 20:45:02.323407534 +0000 UTC m=+1830.613064064" Nov 02 20:45:02.432204 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.432142 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" podStartSLOduration=2.432122381 podStartE2EDuration="2.432122381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:02.430239411 +0000 UTC m=+1830.719895947" watchObservedRunningTime="2026-11-02 20:45:02.432122381 +0000 UTC m=+1830.721778910" Nov 02 20:45:02.544166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.544045 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" podStartSLOduration=2.544030412 podStartE2EDuration="2.544030412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:02.541643473 +0000 UTC m=+1830.831299998" watchObservedRunningTime="2026-11-02 20:45:02.544030412 +0000 UTC m=+1830.833686939" Nov 02 20:45:02.651721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.651681 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" podStartSLOduration=2.651668029 podStartE2EDuration="2.651668029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:02.649630842 +0000 UTC m=+1830.939287371" watchObservedRunningTime="2026-11-02 20:45:02.651668029 +0000 UTC m=+1830.941324557" Nov 02 20:45:02.763970 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.763921 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" podStartSLOduration=2.763907674 podStartE2EDuration="2.763907674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:02.760641681 +0000 UTC m=+1831.050298205" watchObservedRunningTime="2026-11-02 20:45:02.763907674 +0000 UTC m=+1831.053564202" Nov 02 20:45:02.902946 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:02.902909 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:03.121219 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.121188 4971 generic.go:356] "Generic (PLEG): container finished" podID="45bc19b5-e425-4cc1-85b1-2b0f25751cfa" containerID="ea57032d905e68157da162e53f565c66a9a2ddfab0a798242940975d61fbba2c" exitCode=60 Nov 02 20:45:03.121361 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.121270 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" event={"ID":"45bc19b5-e425-4cc1-85b1-2b0f25751cfa","Type":"ContainerDied","Data":"ea57032d905e68157da162e53f565c66a9a2ddfab0a798242940975d61fbba2c"} Nov 02 20:45:03.121489 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.121464 4971 scope.go:122] "RemoveContainer" containerID="ea57032d905e68157da162e53f565c66a9a2ddfab0a798242940975d61fbba2c" Nov 02 20:45:03.123213 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.123196 4971 generic.go:356] "Generic (PLEG): container finished" podID="7d07f1a3-680b-46aa-9a1a-2f7c0911d411" containerID="de3cd1307f053c737fe7a5430204d93c750a6b6da709b2c1509fe5db3172a448" exitCode=60 Nov 02 20:45:03.123310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.123285 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" event={"ID":"7d07f1a3-680b-46aa-9a1a-2f7c0911d411","Type":"ContainerDied","Data":"de3cd1307f053c737fe7a5430204d93c750a6b6da709b2c1509fe5db3172a448"} Nov 02 20:45:03.123538 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.123519 4971 scope.go:122] "RemoveContainer" containerID="de3cd1307f053c737fe7a5430204d93c750a6b6da709b2c1509fe5db3172a448" Nov 02 20:45:03.127123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.126743 4971 generic.go:356] "Generic (PLEG): container finished" podID="c5cb9679-123e-4d5b-9452-82ae0fac9fe7" containerID="ee4d3f5a11a67132fb2a903be5b7aaf4859db49491e765c82d79866d595a1632" exitCode=60 Nov 02 20:45:03.127123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.126786 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" event={"ID":"c5cb9679-123e-4d5b-9452-82ae0fac9fe7","Type":"ContainerDied","Data":"ee4d3f5a11a67132fb2a903be5b7aaf4859db49491e765c82d79866d595a1632"} Nov 02 20:45:03.127461 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.127306 4971 scope.go:122] "RemoveContainer" containerID="ee4d3f5a11a67132fb2a903be5b7aaf4859db49491e765c82d79866d595a1632" Nov 02 20:45:03.128765 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.128743 4971 generic.go:356] "Generic (PLEG): container finished" podID="37c9b65e-f647-4752-8b1e-ab04fb5257ec" containerID="050b1f427c968350eefa8cc242b00eca1e896a7da4f7a7c0e7bfe71e16a7a642" exitCode=60 Nov 02 20:45:03.128868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.128828 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" event={"ID":"37c9b65e-f647-4752-8b1e-ab04fb5257ec","Type":"ContainerDied","Data":"050b1f427c968350eefa8cc242b00eca1e896a7da4f7a7c0e7bfe71e16a7a642"} Nov 02 20:45:03.129050 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.129029 4971 scope.go:122] "RemoveContainer" containerID="050b1f427c968350eefa8cc242b00eca1e896a7da4f7a7c0e7bfe71e16a7a642" Nov 02 20:45:03.130358 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.130340 4971 generic.go:356] "Generic (PLEG): container finished" podID="b0bf8d79-25c9-453a-8395-ca13f8f03d16" containerID="67575b0e9fc743d9c52da9001e99d43c74ef82d05bedfb8073ff794a067caa79" exitCode=60 Nov 02 20:45:03.130465 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.130398 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" event={"ID":"b0bf8d79-25c9-453a-8395-ca13f8f03d16","Type":"ContainerDied","Data":"67575b0e9fc743d9c52da9001e99d43c74ef82d05bedfb8073ff794a067caa79"} Nov 02 20:45:03.130582 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.130568 4971 scope.go:122] "RemoveContainer" containerID="67575b0e9fc743d9c52da9001e99d43c74ef82d05bedfb8073ff794a067caa79" Nov 02 20:45:03.132044 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.132017 4971 generic.go:356] "Generic (PLEG): container finished" podID="54901b53-c9d1-43c1-a587-63845bb341cc" containerID="701cf425b6fe185b1f03f496011ca502cd4d08d3151c0e757a883041edb63ee8" exitCode=60 Nov 02 20:45:03.132159 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.132112 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" event={"ID":"54901b53-c9d1-43c1-a587-63845bb341cc","Type":"ContainerDied","Data":"701cf425b6fe185b1f03f496011ca502cd4d08d3151c0e757a883041edb63ee8"} Nov 02 20:45:03.132307 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.132290 4971 scope.go:122] "RemoveContainer" containerID="701cf425b6fe185b1f03f496011ca502cd4d08d3151c0e757a883041edb63ee8" Nov 02 20:45:03.134191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.134159 4971 generic.go:356] "Generic (PLEG): container finished" podID="6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" containerID="bbf7aad5658555589bcbedd2236536a068e20df67961ab56e9204de392d28af5" exitCode=60 Nov 02 20:45:03.134257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.134235 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" event={"ID":"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5","Type":"ContainerDied","Data":"bbf7aad5658555589bcbedd2236536a068e20df67961ab56e9204de392d28af5"} Nov 02 20:45:03.134471 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.134454 4971 scope.go:122] "RemoveContainer" containerID="bbf7aad5658555589bcbedd2236536a068e20df67961ab56e9204de392d28af5" Nov 02 20:45:03.135806 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.135784 4971 generic.go:356] "Generic (PLEG): container finished" podID="5b7c43be-2b76-4027-8a05-24c8b6bedea5" containerID="79e54665e20f64a8b59d9cb1bf68175658c558de7e878853e7d78f31f7ac83a6" exitCode=60 Nov 02 20:45:03.135893 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.135815 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" event={"ID":"5b7c43be-2b76-4027-8a05-24c8b6bedea5","Type":"ContainerDied","Data":"79e54665e20f64a8b59d9cb1bf68175658c558de7e878853e7d78f31f7ac83a6"} Nov 02 20:45:03.136010 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:03.135998 4971 scope.go:122] "RemoveContainer" containerID="79e54665e20f64a8b59d9cb1bf68175658c558de7e878853e7d78f31f7ac83a6" Nov 02 20:45:03.355888 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:03.355855 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:45:04.146186 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:04.146152 4971 generic.go:356] "Generic (PLEG): container finished" podID="7d07f1a3-680b-46aa-9a1a-2f7c0911d411" containerID="ae4802c871ded2d2a74192fc53f5c8c641d0197831605e40a66a066290a503a8" exitCode=60 Nov 02 20:45:04.146529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:04.146232 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" event={"ID":"7d07f1a3-680b-46aa-9a1a-2f7c0911d411","Type":"ContainerDied","Data":"ae4802c871ded2d2a74192fc53f5c8c641d0197831605e40a66a066290a503a8"} Nov 02 20:45:04.146529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:04.146299 4971 scope.go:122] "RemoveContainer" containerID="de3cd1307f053c737fe7a5430204d93c750a6b6da709b2c1509fe5db3172a448" Nov 02 20:45:04.146644 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:04.146571 4971 scope.go:122] "RemoveContainer" containerID="ae4802c871ded2d2a74192fc53f5c8c641d0197831605e40a66a066290a503a8" Nov 02 20:45:04.146954 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:04.146931 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf_odh-ai-gateway-infra(7d07f1a3-680b-46aa-9a1a-2f7c0911d411)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" podUID="7d07f1a3-680b-46aa-9a1a-2f7c0911d411" Nov 02 20:45:04.148469 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:04.148449 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" event={"ID":"c5cb9679-123e-4d5b-9452-82ae0fac9fe7","Type":"ContainerStarted","Data":"cd79aa4d775b0bb4cb7797ba36f24ab36d0303b71650c76f26cf422927fdb908"} Nov 02 20:45:04.150477 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:04.150457 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" event={"ID":"37c9b65e-f647-4752-8b1e-ab04fb5257ec","Type":"ContainerStarted","Data":"145e2edfd8b8a6e1e62079fc20240ffefb5ab84a2848a2654a183029224461c7"} Nov 02 20:45:04.152398 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:04.152379 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" event={"ID":"b0bf8d79-25c9-453a-8395-ca13f8f03d16","Type":"ContainerStarted","Data":"3a865135828834d78368ae307a311752666d4624c1b7208bbd71844a560d943f"} Nov 02 20:45:04.154364 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:04.154336 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" event={"ID":"54901b53-c9d1-43c1-a587-63845bb341cc","Type":"ContainerStarted","Data":"c0c91ba6424f5843b068ab1d40265dded9c6b06249de844b3e21f66b4225156e"} Nov 02 20:45:04.156301 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:04.156283 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" event={"ID":"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5","Type":"ContainerStarted","Data":"d6854a0edce336c48680ec5c05cd43bc0f267531f8f63046f7b6affc9fe98708"} Nov 02 20:45:04.158057 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:04.158037 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" event={"ID":"5b7c43be-2b76-4027-8a05-24c8b6bedea5","Type":"ContainerStarted","Data":"7f58b9b205547599ce709064afbcc067f1039a3945360d284af7c7ccbbf8c872"} Nov 02 20:45:04.159779 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:04.159759 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" event={"ID":"45bc19b5-e425-4cc1-85b1-2b0f25751cfa","Type":"ContainerStarted","Data":"ea8bac6ac85655347858233a9ea5d541a9954ef35bd33eac8176ea2970d2d8ad"} Nov 02 20:45:04.369849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:04.369788 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" podStartSLOduration=4.369772698 podStartE2EDuration="4.369772698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:04.365974853 +0000 UTC m=+1832.655631372" watchObservedRunningTime="2026-11-02 20:45:04.369772698 +0000 UTC m=+1832.659429221" Nov 02 20:45:05.169771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.169745 4971 scope.go:122] "RemoveContainer" containerID="ae4802c871ded2d2a74192fc53f5c8c641d0197831605e40a66a066290a503a8" Nov 02 20:45:05.170269 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:05.170007 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf_odh-ai-gateway-infra(7d07f1a3-680b-46aa-9a1a-2f7c0911d411)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" podUID="7d07f1a3-680b-46aa-9a1a-2f7c0911d411" Nov 02 20:45:05.171624 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.171599 4971 generic.go:356] "Generic (PLEG): container finished" podID="c5cb9679-123e-4d5b-9452-82ae0fac9fe7" containerID="cd79aa4d775b0bb4cb7797ba36f24ab36d0303b71650c76f26cf422927fdb908" exitCode=60 Nov 02 20:45:05.171734 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.171644 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" event={"ID":"c5cb9679-123e-4d5b-9452-82ae0fac9fe7","Type":"ContainerDied","Data":"cd79aa4d775b0bb4cb7797ba36f24ab36d0303b71650c76f26cf422927fdb908"} Nov 02 20:45:05.171734 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.171698 4971 scope.go:122] "RemoveContainer" containerID="ee4d3f5a11a67132fb2a903be5b7aaf4859db49491e765c82d79866d595a1632" Nov 02 20:45:05.171919 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.171904 4971 scope.go:122] "RemoveContainer" containerID="cd79aa4d775b0bb4cb7797ba36f24ab36d0303b71650c76f26cf422927fdb908" Nov 02 20:45:05.172207 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:05.172185 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr_odh-ai-gateway-infra(c5cb9679-123e-4d5b-9452-82ae0fac9fe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" podUID="c5cb9679-123e-4d5b-9452-82ae0fac9fe7" Nov 02 20:45:05.173844 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.173820 4971 generic.go:356] "Generic (PLEG): container finished" podID="37c9b65e-f647-4752-8b1e-ab04fb5257ec" containerID="145e2edfd8b8a6e1e62079fc20240ffefb5ab84a2848a2654a183029224461c7" exitCode=60 Nov 02 20:45:05.173950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.173912 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" event={"ID":"37c9b65e-f647-4752-8b1e-ab04fb5257ec","Type":"ContainerDied","Data":"145e2edfd8b8a6e1e62079fc20240ffefb5ab84a2848a2654a183029224461c7"} Nov 02 20:45:05.174129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.174114 4971 scope.go:122] "RemoveContainer" containerID="145e2edfd8b8a6e1e62079fc20240ffefb5ab84a2848a2654a183029224461c7" Nov 02 20:45:05.174386 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:05.174356 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc_odh-ai-gateway-infra(37c9b65e-f647-4752-8b1e-ab04fb5257ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" podUID="37c9b65e-f647-4752-8b1e-ab04fb5257ec" Nov 02 20:45:05.175806 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.175782 4971 generic.go:356] "Generic (PLEG): container finished" podID="b0bf8d79-25c9-453a-8395-ca13f8f03d16" containerID="3a865135828834d78368ae307a311752666d4624c1b7208bbd71844a560d943f" exitCode=60 Nov 02 20:45:05.175893 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.175856 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" event={"ID":"b0bf8d79-25c9-453a-8395-ca13f8f03d16","Type":"ContainerDied","Data":"3a865135828834d78368ae307a311752666d4624c1b7208bbd71844a560d943f"} Nov 02 20:45:05.176134 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.176117 4971 scope.go:122] "RemoveContainer" containerID="3a865135828834d78368ae307a311752666d4624c1b7208bbd71844a560d943f" Nov 02 20:45:05.181774 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:05.176343 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl_odh-ai-gateway-infra(b0bf8d79-25c9-453a-8395-ca13f8f03d16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" podUID="b0bf8d79-25c9-453a-8395-ca13f8f03d16" Nov 02 20:45:05.183534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.183510 4971 generic.go:356] "Generic (PLEG): container finished" podID="54901b53-c9d1-43c1-a587-63845bb341cc" containerID="c0c91ba6424f5843b068ab1d40265dded9c6b06249de844b3e21f66b4225156e" exitCode=60 Nov 02 20:45:05.183613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.183591 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" event={"ID":"54901b53-c9d1-43c1-a587-63845bb341cc","Type":"ContainerDied","Data":"c0c91ba6424f5843b068ab1d40265dded9c6b06249de844b3e21f66b4225156e"} Nov 02 20:45:05.183777 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.183763 4971 scope.go:122] "RemoveContainer" containerID="c0c91ba6424f5843b068ab1d40265dded9c6b06249de844b3e21f66b4225156e" Nov 02 20:45:05.183984 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:05.183968 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c_odh-ai-gateway-infra(54901b53-c9d1-43c1-a587-63845bb341cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" podUID="54901b53-c9d1-43c1-a587-63845bb341cc" Nov 02 20:45:05.186279 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.186259 4971 generic.go:356] "Generic (PLEG): container finished" podID="6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" containerID="d6854a0edce336c48680ec5c05cd43bc0f267531f8f63046f7b6affc9fe98708" exitCode=60 Nov 02 20:45:05.186375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.186326 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" event={"ID":"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5","Type":"ContainerDied","Data":"d6854a0edce336c48680ec5c05cd43bc0f267531f8f63046f7b6affc9fe98708"} Nov 02 20:45:05.186503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.186489 4971 scope.go:122] "RemoveContainer" containerID="d6854a0edce336c48680ec5c05cd43bc0f267531f8f63046f7b6affc9fe98708" Nov 02 20:45:05.186721 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:05.186703 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2_odh-ai-gateway-infra(6f52b903-f5e5-4ade-b79e-12c5a1f79ff5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" podUID="6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" Nov 02 20:45:05.188210 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.188187 4971 generic.go:356] "Generic (PLEG): container finished" podID="5b7c43be-2b76-4027-8a05-24c8b6bedea5" containerID="7f58b9b205547599ce709064afbcc067f1039a3945360d284af7c7ccbbf8c872" exitCode=60 Nov 02 20:45:05.188320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.188269 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" event={"ID":"5b7c43be-2b76-4027-8a05-24c8b6bedea5","Type":"ContainerDied","Data":"7f58b9b205547599ce709064afbcc067f1039a3945360d284af7c7ccbbf8c872"} Nov 02 20:45:05.188453 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.188436 4971 scope.go:122] "RemoveContainer" containerID="7f58b9b205547599ce709064afbcc067f1039a3945360d284af7c7ccbbf8c872" Nov 02 20:45:05.188703 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:05.188681 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47_odh-ai-gateway-infra(5b7c43be-2b76-4027-8a05-24c8b6bedea5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" podUID="5b7c43be-2b76-4027-8a05-24c8b6bedea5" Nov 02 20:45:05.188888 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.188854 4971 scope.go:122] "RemoveContainer" containerID="050b1f427c968350eefa8cc242b00eca1e896a7da4f7a7c0e7bfe71e16a7a642" Nov 02 20:45:05.190222 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.190199 4971 generic.go:356] "Generic (PLEG): container finished" podID="45bc19b5-e425-4cc1-85b1-2b0f25751cfa" containerID="ea8bac6ac85655347858233a9ea5d541a9954ef35bd33eac8176ea2970d2d8ad" exitCode=60 Nov 02 20:45:05.190306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.190281 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" event={"ID":"45bc19b5-e425-4cc1-85b1-2b0f25751cfa","Type":"ContainerDied","Data":"ea8bac6ac85655347858233a9ea5d541a9954ef35bd33eac8176ea2970d2d8ad"} Nov 02 20:45:05.190494 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.190481 4971 scope.go:122] "RemoveContainer" containerID="ea8bac6ac85655347858233a9ea5d541a9954ef35bd33eac8176ea2970d2d8ad" Nov 02 20:45:05.190688 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:05.190666 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x_odh-ai-gateway-infra(45bc19b5-e425-4cc1-85b1-2b0f25751cfa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" podUID="45bc19b5-e425-4cc1-85b1-2b0f25751cfa" Nov 02 20:45:05.202313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.202296 4971 scope.go:122] "RemoveContainer" containerID="67575b0e9fc743d9c52da9001e99d43c74ef82d05bedfb8073ff794a067caa79" Nov 02 20:45:05.215941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.215921 4971 scope.go:122] "RemoveContainer" containerID="701cf425b6fe185b1f03f496011ca502cd4d08d3151c0e757a883041edb63ee8" Nov 02 20:45:05.229254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.229238 4971 scope.go:122] "RemoveContainer" containerID="bbf7aad5658555589bcbedd2236536a068e20df67961ab56e9204de392d28af5" Nov 02 20:45:05.241896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.241873 4971 scope.go:122] "RemoveContainer" containerID="79e54665e20f64a8b59d9cb1bf68175658c558de7e878853e7d78f31f7ac83a6" Nov 02 20:45:05.255181 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:05.255160 4971 scope.go:122] "RemoveContainer" containerID="ea57032d905e68157da162e53f565c66a9a2ddfab0a798242940975d61fbba2c" Nov 02 20:45:06.198930 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:06.198902 4971 scope.go:122] "RemoveContainer" containerID="ea8bac6ac85655347858233a9ea5d541a9954ef35bd33eac8176ea2970d2d8ad" Nov 02 20:45:06.199382 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:06.199245 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x_odh-ai-gateway-infra(45bc19b5-e425-4cc1-85b1-2b0f25751cfa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" podUID="45bc19b5-e425-4cc1-85b1-2b0f25751cfa" Nov 02 20:45:06.200579 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:06.200552 4971 scope.go:122] "RemoveContainer" containerID="cd79aa4d775b0bb4cb7797ba36f24ab36d0303b71650c76f26cf422927fdb908" Nov 02 20:45:06.200839 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:06.200820 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr_odh-ai-gateway-infra(c5cb9679-123e-4d5b-9452-82ae0fac9fe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" podUID="c5cb9679-123e-4d5b-9452-82ae0fac9fe7" Nov 02 20:45:06.202157 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:06.202138 4971 scope.go:122] "RemoveContainer" containerID="145e2edfd8b8a6e1e62079fc20240ffefb5ab84a2848a2654a183029224461c7" Nov 02 20:45:06.202381 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:06.202357 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc_odh-ai-gateway-infra(37c9b65e-f647-4752-8b1e-ab04fb5257ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" podUID="37c9b65e-f647-4752-8b1e-ab04fb5257ec" Nov 02 20:45:06.203748 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:06.203729 4971 scope.go:122] "RemoveContainer" containerID="3a865135828834d78368ae307a311752666d4624c1b7208bbd71844a560d943f" Nov 02 20:45:06.203985 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:06.203963 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl_odh-ai-gateway-infra(b0bf8d79-25c9-453a-8395-ca13f8f03d16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" podUID="b0bf8d79-25c9-453a-8395-ca13f8f03d16" Nov 02 20:45:06.205241 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:06.205225 4971 scope.go:122] "RemoveContainer" containerID="c0c91ba6424f5843b068ab1d40265dded9c6b06249de844b3e21f66b4225156e" Nov 02 20:45:06.205438 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:06.205415 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c_odh-ai-gateway-infra(54901b53-c9d1-43c1-a587-63845bb341cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" podUID="54901b53-c9d1-43c1-a587-63845bb341cc" Nov 02 20:45:06.206858 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:06.206842 4971 scope.go:122] "RemoveContainer" containerID="d6854a0edce336c48680ec5c05cd43bc0f267531f8f63046f7b6affc9fe98708" Nov 02 20:45:06.207065 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:06.207047 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2_odh-ai-gateway-infra(6f52b903-f5e5-4ade-b79e-12c5a1f79ff5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" podUID="6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" Nov 02 20:45:06.208330 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:06.208313 4971 scope.go:122] "RemoveContainer" containerID="7f58b9b205547599ce709064afbcc067f1039a3945360d284af7c7ccbbf8c872" Nov 02 20:45:06.208496 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:06.208480 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47_odh-ai-gateway-infra(5b7c43be-2b76-4027-8a05-24c8b6bedea5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" podUID="5b7c43be-2b76-4027-8a05-24c8b6bedea5" Nov 02 20:45:07.231004 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.230969 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v"] Nov 02 20:45:07.344639 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.344610 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j"] Nov 02 20:45:07.598183 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.598150 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" Nov 02 20:45:07.719791 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.719760 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:07.721169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.721139 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96107d44-a6a7-4af1-840b-706c09bf59a1-kube-api-access-p55lw\" (UniqueName: \"kubernetes.io/projected/96107d44-a6a7-4af1-840b-706c09bf59a1-kube-api-access-p55lw\") pod \"96107d44-a6a7-4af1-840b-706c09bf59a1\" (UID: \"96107d44-a6a7-4af1-840b-706c09bf59a1\") " Nov 02 20:45:07.721308 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.721194 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96107d44-a6a7-4af1-840b-706c09bf59a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96107d44-a6a7-4af1-840b-706c09bf59a1-plugins-config-volume\") pod \"96107d44-a6a7-4af1-840b-706c09bf59a1\" (UID: \"96107d44-a6a7-4af1-840b-706c09bf59a1\") " Nov 02 20:45:07.722021 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.721992 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96107d44-a6a7-4af1-840b-706c09bf59a1-plugins-config-volume" pod "96107d44-a6a7-4af1-840b-706c09bf59a1" (UID: "96107d44-a6a7-4af1-840b-706c09bf59a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:07.733358 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.733329 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96107d44-a6a7-4af1-840b-706c09bf59a1-kube-api-access-p55lw" pod "96107d44-a6a7-4af1-840b-706c09bf59a1" (UID: "96107d44-a6a7-4af1-840b-706c09bf59a1"). InnerVolumeSpecName "kube-api-access-p55lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:07.760174 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.760146 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" Nov 02 20:45:07.822875 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.822776 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p55lw\" (UniqueName: \"kubernetes.io/projected/96107d44-a6a7-4af1-840b-706c09bf59a1-kube-api-access-p55lw\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:07.822875 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.822806 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96107d44-a6a7-4af1-840b-706c09bf59a1-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:07.924221 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.924186 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e1c40c-fa5a-4707-8d39-24e1d57b0d91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e1c40c-fa5a-4707-8d39-24e1d57b0d91-plugins-config-volume\") pod \"c4e1c40c-fa5a-4707-8d39-24e1d57b0d91\" (UID: \"c4e1c40c-fa5a-4707-8d39-24e1d57b0d91\") " Nov 02 20:45:07.924424 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.924397 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e1c40c-fa5a-4707-8d39-24e1d57b0d91-kube-api-access-bnfp8\" (UniqueName: \"kubernetes.io/projected/c4e1c40c-fa5a-4707-8d39-24e1d57b0d91-kube-api-access-bnfp8\") pod \"c4e1c40c-fa5a-4707-8d39-24e1d57b0d91\" (UID: \"c4e1c40c-fa5a-4707-8d39-24e1d57b0d91\") " Nov 02 20:45:07.924675 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.924645 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e1c40c-fa5a-4707-8d39-24e1d57b0d91-plugins-config-volume" pod "c4e1c40c-fa5a-4707-8d39-24e1d57b0d91" (UID: "c4e1c40c-fa5a-4707-8d39-24e1d57b0d91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:07.924977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.924957 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e1c40c-fa5a-4707-8d39-24e1d57b0d91-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:07.926898 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.926875 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e1c40c-fa5a-4707-8d39-24e1d57b0d91-kube-api-access-bnfp8" pod "c4e1c40c-fa5a-4707-8d39-24e1d57b0d91" (UID: "c4e1c40c-fa5a-4707-8d39-24e1d57b0d91"). InnerVolumeSpecName "kube-api-access-bnfp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:07.964705 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:07.964676 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:08.026660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.026628 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnfp8\" (UniqueName: \"kubernetes.io/projected/c4e1c40c-fa5a-4707-8d39-24e1d57b0d91-kube-api-access-bnfp8\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:08.226039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.226008 4971 generic.go:356] "Generic (PLEG): container finished" podID="96107d44-a6a7-4af1-840b-706c09bf59a1" containerID="ee735e00b0479fb65c40adfe1f0b9778dcebaeb39ba74337739d6e1288e4dfe7" exitCode=0 Nov 02 20:45:08.226245 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.226053 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" Nov 02 20:45:08.226245 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.226107 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" event={"ID":"96107d44-a6a7-4af1-840b-706c09bf59a1","Type":"ContainerDied","Data":"ee735e00b0479fb65c40adfe1f0b9778dcebaeb39ba74337739d6e1288e4dfe7"} Nov 02 20:45:08.226245 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.226165 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42" event={"ID":"96107d44-a6a7-4af1-840b-706c09bf59a1","Type":"ContainerDied","Data":"82d58eee8f3c3dc18fea2690c9c10003a9e309de7110e141c209307719b1608e"} Nov 02 20:45:08.226245 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.226198 4971 scope.go:122] "RemoveContainer" containerID="ee735e00b0479fb65c40adfe1f0b9778dcebaeb39ba74337739d6e1288e4dfe7" Nov 02 20:45:08.228028 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.228004 4971 generic.go:356] "Generic (PLEG): container finished" podID="c4e1c40c-fa5a-4707-8d39-24e1d57b0d91" containerID="2ba174f8e1d3678c5a27e4fc7a531d5ecfc0d151e194eecd4a69d18c95e21b19" exitCode=0 Nov 02 20:45:08.228263 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.228050 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" Nov 02 20:45:08.228263 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.228055 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" event={"ID":"c4e1c40c-fa5a-4707-8d39-24e1d57b0d91","Type":"ContainerDied","Data":"2ba174f8e1d3678c5a27e4fc7a531d5ecfc0d151e194eecd4a69d18c95e21b19"} Nov 02 20:45:08.228263 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.228131 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp" event={"ID":"c4e1c40c-fa5a-4707-8d39-24e1d57b0d91","Type":"ContainerDied","Data":"fc257d7ac591deaa64f2ef82de1a48647d5fe6ea1b8be6b9f18ff6c7c5823c37"} Nov 02 20:45:08.237470 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.237455 4971 scope.go:122] "RemoveContainer" containerID="ee735e00b0479fb65c40adfe1f0b9778dcebaeb39ba74337739d6e1288e4dfe7" Nov 02 20:45:08.237776 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:08.237715 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee735e00b0479fb65c40adfe1f0b9778dcebaeb39ba74337739d6e1288e4dfe7\": container with ID starting with ee735e00b0479fb65c40adfe1f0b9778dcebaeb39ba74337739d6e1288e4dfe7 not found: ID does not exist" containerID="ee735e00b0479fb65c40adfe1f0b9778dcebaeb39ba74337739d6e1288e4dfe7" Nov 02 20:45:08.237776 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.237739 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee735e00b0479fb65c40adfe1f0b9778dcebaeb39ba74337739d6e1288e4dfe7"} err="failed to get container status \"ee735e00b0479fb65c40adfe1f0b9778dcebaeb39ba74337739d6e1288e4dfe7\": rpc error: code = NotFound desc = could not find container \"ee735e00b0479fb65c40adfe1f0b9778dcebaeb39ba74337739d6e1288e4dfe7\": container with ID starting with ee735e00b0479fb65c40adfe1f0b9778dcebaeb39ba74337739d6e1288e4dfe7 not found: ID does not exist" Nov 02 20:45:08.237776 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.237756 4971 scope.go:122] "RemoveContainer" containerID="2ba174f8e1d3678c5a27e4fc7a531d5ecfc0d151e194eecd4a69d18c95e21b19" Nov 02 20:45:08.247556 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.247541 4971 scope.go:122] "RemoveContainer" containerID="2ba174f8e1d3678c5a27e4fc7a531d5ecfc0d151e194eecd4a69d18c95e21b19" Nov 02 20:45:08.247807 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:08.247789 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba174f8e1d3678c5a27e4fc7a531d5ecfc0d151e194eecd4a69d18c95e21b19\": container with ID starting with 2ba174f8e1d3678c5a27e4fc7a531d5ecfc0d151e194eecd4a69d18c95e21b19 not found: ID does not exist" containerID="2ba174f8e1d3678c5a27e4fc7a531d5ecfc0d151e194eecd4a69d18c95e21b19" Nov 02 20:45:08.247868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.247815 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba174f8e1d3678c5a27e4fc7a531d5ecfc0d151e194eecd4a69d18c95e21b19"} err="failed to get container status \"2ba174f8e1d3678c5a27e4fc7a531d5ecfc0d151e194eecd4a69d18c95e21b19\": rpc error: code = NotFound desc = could not find container \"2ba174f8e1d3678c5a27e4fc7a531d5ecfc0d151e194eecd4a69d18c95e21b19\": container with ID starting with 2ba174f8e1d3678c5a27e4fc7a531d5ecfc0d151e194eecd4a69d18c95e21b19 not found: ID does not exist" Nov 02 20:45:08.337930 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.337897 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:08.391170 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.391142 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp"] Nov 02 20:45:08.395557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.395536 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-c49693-76bb546599-z4xvp"] Nov 02 20:45:08.522487 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.522420 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42"] Nov 02 20:45:08.577643 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:08.577612 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-c49693-5b867586b8-skz42"] Nov 02 20:45:09.866895 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:09.866854 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96107d44-a6a7-4af1-840b-706c09bf59a1" path="/var/lib/kubelet/pods/96107d44-a6a7-4af1-840b-706c09bf59a1/volumes" Nov 02 20:45:09.867447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:09.867426 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e1c40c-fa5a-4707-8d39-24e1d57b0d91" path="/var/lib/kubelet/pods/c4e1c40c-fa5a-4707-8d39-24e1d57b0d91/volumes" Nov 02 20:45:12.232256 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:12.232173 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" podUID="2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3" containerName="payload-processing" containerID="cri-o://329c36247d67f0b919b5878124ef7f872924c5ff9564961b3f0b0931a3818a43" gracePeriod=25 Nov 02 20:45:12.345169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:12.345121 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" podUID="f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c" containerName="payload-pre-processing" containerID="cri-o://07b7d016cb65a3c2559cd8bcee7e975aa8da695985f1a5ee49aae19d7d18beb3" gracePeriod=25 Nov 02 20:45:12.636171 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:12.636130 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" podUID="2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:45:13.412446 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:13.412409 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:15.147690 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.147642 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" containerID="cri-o://3a776f5ba0e19fe7aada11d831ca6de2e2d921574c0e4d7a751487a5fddf0725" gracePeriod=45 Nov 02 20:45:15.148229 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.148161 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" containerID="cri-o://9383f2d99c9f96b84b23fb1f07e8956306e14b07c7b692470327c2a9230e409e" gracePeriod=45 Nov 02 20:45:15.290570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.290537 4971 generic.go:356] "Generic (PLEG): container finished" podID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerID="9383f2d99c9f96b84b23fb1f07e8956306e14b07c7b692470327c2a9230e409e" exitCode=0 Nov 02 20:45:15.290814 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.290787 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" event={"ID":"9bf3f8ce-7c86-4fa5-ac99-bf2532808882","Type":"ContainerDied","Data":"9383f2d99c9f96b84b23fb1f07e8956306e14b07c7b692470327c2a9230e409e"} Nov 02 20:45:15.292582 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.292557 4971 generic.go:356] "Generic (PLEG): container finished" podID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerID="3a776f5ba0e19fe7aada11d831ca6de2e2d921574c0e4d7a751487a5fddf0725" exitCode=0 Nov 02 20:45:15.292676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.292590 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" event={"ID":"10dd59b7-c045-4b1a-a27e-b7034e0b653d","Type":"ContainerDied","Data":"3a776f5ba0e19fe7aada11d831ca6de2e2d921574c0e4d7a751487a5fddf0725"} Nov 02 20:45:15.514457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.514429 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:45:15.516699 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.516627 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:45:15.579121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.577979 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-kube-api-access-t5kz2\" (UniqueName: \"kubernetes.io/projected/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-kube-api-access-t5kz2\") pod \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " Nov 02 20:45:15.579121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.578062 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-model-cache\") pod \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " Nov 02 20:45:15.579121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.578136 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-home\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-home\") pod \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " Nov 02 20:45:15.579121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.578184 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-tmp-dir\") pod \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " Nov 02 20:45:15.579121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.578231 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-model-cache\") pod \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " Nov 02 20:45:15.579121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.578285 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-tmp-dir\") pod \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " Nov 02 20:45:15.579121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.578340 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-dshm\") pod \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " Nov 02 20:45:15.579121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.578367 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-dshm\") pod \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " Nov 02 20:45:15.579121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.578408 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10dd59b7-c045-4b1a-a27e-b7034e0b653d-kube-api-access-rtzbp\" (UniqueName: \"kubernetes.io/projected/10dd59b7-c045-4b1a-a27e-b7034e0b653d-kube-api-access-rtzbp\") pod \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " Nov 02 20:45:15.579121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.578456 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-home\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-home\") pod \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " Nov 02 20:45:15.579121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.578590 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-tls-certs\") pod \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\" (UID: \"9bf3f8ce-7c86-4fa5-ac99-bf2532808882\") " Nov 02 20:45:15.579121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.578620 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10dd59b7-c045-4b1a-a27e-b7034e0b653d-tls-certs\" (UniqueName: \"kubernetes.io/secret/10dd59b7-c045-4b1a-a27e-b7034e0b653d-tls-certs\") pod \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\" (UID: \"10dd59b7-c045-4b1a-a27e-b7034e0b653d\") " Nov 02 20:45:15.579121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.578725 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-tmp-dir" pod "9bf3f8ce-7c86-4fa5-ac99-bf2532808882" (UID: "9bf3f8ce-7c86-4fa5-ac99-bf2532808882"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:15.579785 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.579189 4971 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-tmp-dir\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.580442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.580413 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-home" pod "9bf3f8ce-7c86-4fa5-ac99-bf2532808882" (UID: "9bf3f8ce-7c86-4fa5-ac99-bf2532808882"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:15.580782 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.580613 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-tmp-dir" pod "10dd59b7-c045-4b1a-a27e-b7034e0b653d" (UID: "10dd59b7-c045-4b1a-a27e-b7034e0b653d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:15.581328 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.580016 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-model-cache" pod "10dd59b7-c045-4b1a-a27e-b7034e0b653d" (UID: "10dd59b7-c045-4b1a-a27e-b7034e0b653d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:15.581763 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.581656 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-model-cache" pod "9bf3f8ce-7c86-4fa5-ac99-bf2532808882" (UID: "9bf3f8ce-7c86-4fa5-ac99-bf2532808882"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:15.581763 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.581738 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-home" pod "10dd59b7-c045-4b1a-a27e-b7034e0b653d" (UID: "10dd59b7-c045-4b1a-a27e-b7034e0b653d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:15.583007 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.582983 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10dd59b7-c045-4b1a-a27e-b7034e0b653d-tls-certs" pod "10dd59b7-c045-4b1a-a27e-b7034e0b653d" (UID: "10dd59b7-c045-4b1a-a27e-b7034e0b653d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:45:15.583848 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.583799 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-tls-certs" pod "9bf3f8ce-7c86-4fa5-ac99-bf2532808882" (UID: "9bf3f8ce-7c86-4fa5-ac99-bf2532808882"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:45:15.584990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.584964 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-dshm" pod "10dd59b7-c045-4b1a-a27e-b7034e0b653d" (UID: "10dd59b7-c045-4b1a-a27e-b7034e0b653d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:15.585299 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.585276 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-dshm" pod "9bf3f8ce-7c86-4fa5-ac99-bf2532808882" (UID: "9bf3f8ce-7c86-4fa5-ac99-bf2532808882"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:15.586779 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.586756 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10dd59b7-c045-4b1a-a27e-b7034e0b653d-kube-api-access-rtzbp" pod "10dd59b7-c045-4b1a-a27e-b7034e0b653d" (UID: "10dd59b7-c045-4b1a-a27e-b7034e0b653d"). InnerVolumeSpecName "kube-api-access-rtzbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:15.592797 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.592774 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-kube-api-access-t5kz2" pod "9bf3f8ce-7c86-4fa5-ac99-bf2532808882" (UID: "9bf3f8ce-7c86-4fa5-ac99-bf2532808882"). InnerVolumeSpecName "kube-api-access-t5kz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:15.680539 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.680505 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5kz2\" (UniqueName: \"kubernetes.io/projected/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-kube-api-access-t5kz2\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.680539 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.680541 4971 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-model-cache\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.680731 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.680550 4971 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-home\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.680731 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.680559 4971 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-model-cache\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.680731 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.680568 4971 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-tmp-dir\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.680731 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.680579 4971 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-dshm\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.680731 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.680586 4971 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10dd59b7-c045-4b1a-a27e-b7034e0b653d-dshm\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.680731 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.680594 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtzbp\" (UniqueName: \"kubernetes.io/projected/10dd59b7-c045-4b1a-a27e-b7034e0b653d-kube-api-access-rtzbp\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.680731 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.680601 4971 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-home\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.680731 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.680614 4971 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf3f8ce-7c86-4fa5-ac99-bf2532808882-tls-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.680731 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.680625 4971 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10dd59b7-c045-4b1a-a27e-b7034e0b653d-tls-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.859375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.859345 4971 scope.go:122] "RemoveContainer" containerID="7f58b9b205547599ce709064afbcc067f1039a3945360d284af7c7ccbbf8c872" Nov 02 20:45:15.859565 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:15.859543 4971 scope.go:122] "RemoveContainer" containerID="cd79aa4d775b0bb4cb7797ba36f24ab36d0303b71650c76f26cf422927fdb908" Nov 02 20:45:16.157847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.157764 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:16.301973 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.301865 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" Nov 02 20:45:16.301973 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.301891 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw" event={"ID":"10dd59b7-c045-4b1a-a27e-b7034e0b653d","Type":"ContainerDied","Data":"d83094d927e9e5b27133b5b7d6e479abb0bd18b366d4d276a81ce5f40db18d72"} Nov 02 20:45:16.301973 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.301954 4971 scope.go:122] "RemoveContainer" containerID="3a776f5ba0e19fe7aada11d831ca6de2e2d921574c0e4d7a751487a5fddf0725" Nov 02 20:45:16.304135 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.303824 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" event={"ID":"9bf3f8ce-7c86-4fa5-ac99-bf2532808882","Type":"ContainerDied","Data":"6d061230adb7d558d87c3738caca1dfe78721e9c0896018eee65f399fdc0544d"} Nov 02 20:45:16.304135 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.303915 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" Nov 02 20:45:16.308026 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.308003 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" event={"ID":"c5cb9679-123e-4d5b-9452-82ae0fac9fe7","Type":"ContainerStarted","Data":"2a39f079aae82d9d21b7f453768abcca8d1411f6af9a7fda5716f46f68c3bcdb"} Nov 02 20:45:16.315899 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.315881 4971 scope.go:122] "RemoveContainer" containerID="9383f2d99c9f96b84b23fb1f07e8956306e14b07c7b692470327c2a9230e409e" Nov 02 20:45:16.454031 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.453939 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:45:16.550192 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.550109 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:16.568882 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.568848 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2"] Nov 02 20:45:16.572555 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.572527 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bce768/simulator-w4-bce768-kserve-669b8d5956-jv4w2"] Nov 02 20:45:16.674770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.674666 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw"] Nov 02 20:45:16.678732 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.678706 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bce768/distinct-w4-bce768-kserve-c5795f59b-2vdnw"] Nov 02 20:45:16.866158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.864252 4971 scope.go:122] "RemoveContainer" containerID="145e2edfd8b8a6e1e62079fc20240ffefb5ab84a2848a2654a183029224461c7" Nov 02 20:45:16.866158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.865490 4971 scope.go:122] "RemoveContainer" containerID="3a865135828834d78368ae307a311752666d4624c1b7208bbd71844a560d943f" Nov 02 20:45:16.964021 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.963992 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5"] Nov 02 20:45:16.965353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.965331 4971 cpu_manager.go:414] "containerMap: removing container" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" Nov 02 20:45:16.965473 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.965363 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" Nov 02 20:45:16.965473 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.965396 4971 cpu_manager.go:414] "containerMap: removing container" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" Nov 02 20:45:16.965473 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.965402 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" Nov 02 20:45:16.965473 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.965414 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c4e1c40c-fa5a-4707-8d39-24e1d57b0d91" containerName="payload-pre-processing" Nov 02 20:45:16.965473 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.965418 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e1c40c-fa5a-4707-8d39-24e1d57b0d91" containerName="payload-pre-processing" Nov 02 20:45:16.965473 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.965450 4971 cpu_manager.go:414] "containerMap: removing container" podUID="96107d44-a6a7-4af1-840b-706c09bf59a1" containerName="payload-processing" Nov 02 20:45:16.965473 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.965457 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96107d44-a6a7-4af1-840b-706c09bf59a1" containerName="payload-processing" Nov 02 20:45:16.965721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.965556 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" containerName="main" Nov 02 20:45:16.965721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.965569 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e1c40c-fa5a-4707-8d39-24e1d57b0d91" containerName="payload-pre-processing" Nov 02 20:45:16.965721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.965579 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="96107d44-a6a7-4af1-840b-706c09bf59a1" containerName="payload-processing" Nov 02 20:45:16.965721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.965588 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" containerName="main" Nov 02 20:45:16.970717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:16.970698 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.004051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.000228 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-workload-socket\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.004051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.000365 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jkmb\" (UniqueName: \"kubernetes.io/projected/7906ad9c-8297-4513-aaf2-8729d580bfa5-kube-api-access-4jkmb\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.004051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.000536 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-podinfo\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.004051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.000632 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-credential-socket\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.004051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.000670 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-data\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.004051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.000712 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-envoy\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.004051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.000741 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7906ad9c-8297-4513-aaf2-8729d580bfa5-istiod-ca-cert\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.004051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.000762 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-token\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.004051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.000798 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-workload-certs\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.196775 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.026978 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-58c2e0-openshift-default-dockercfg-sv5qn\"" Nov 02 20:45:17.196775 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.041260 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5"] Nov 02 20:45:17.237857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.201026 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jkmb\" (UniqueName: \"kubernetes.io/projected/7906ad9c-8297-4513-aaf2-8729d580bfa5-kube-api-access-4jkmb\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.237857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.201259 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-podinfo\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.237857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.201365 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-credential-socket\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.237857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.201399 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-data\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.237857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.201465 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-envoy\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.237857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.201496 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7906ad9c-8297-4513-aaf2-8729d580bfa5-istiod-ca-cert\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.237857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.201527 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-token\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.237857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.201578 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-workload-certs\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.237857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.201668 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-workload-socket\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.237857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.216353 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:17.237857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.217575 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7906ad9c-8297-4513-aaf2-8729d580bfa5-istiod-ca-cert\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.237857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.218002 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-credential-socket\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.237857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.218355 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-workload-certs\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.237857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.218663 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-data\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.237857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.221459 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-workload-socket\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.285578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.266183 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jkmb\" (UniqueName: \"kubernetes.io/projected/7906ad9c-8297-4513-aaf2-8729d580bfa5-kube-api-access-4jkmb\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.285578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.282772 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-token\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.289978 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.289918 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-envoy\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.291282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.291257 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-podinfo\") pod \"e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.292559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.292536 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:17.367628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.354065 4971 generic.go:356] "Generic (PLEG): container finished" podID="5b7c43be-2b76-4027-8a05-24c8b6bedea5" containerID="a067d8f0fffd8e026b3bd4855cc9cd440f9e301870d46061b568c9695ae4f252" exitCode=60 Nov 02 20:45:17.367628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.354275 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" event={"ID":"5b7c43be-2b76-4027-8a05-24c8b6bedea5","Type":"ContainerDied","Data":"a067d8f0fffd8e026b3bd4855cc9cd440f9e301870d46061b568c9695ae4f252"} Nov 02 20:45:17.367628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.354337 4971 scope.go:122] "RemoveContainer" containerID="7f58b9b205547599ce709064afbcc067f1039a3945360d284af7c7ccbbf8c872" Nov 02 20:45:17.367628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.354814 4971 scope.go:122] "RemoveContainer" containerID="a067d8f0fffd8e026b3bd4855cc9cd440f9e301870d46061b568c9695ae4f252" Nov 02 20:45:17.367628 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:17.355254 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47_odh-ai-gateway-infra(5b7c43be-2b76-4027-8a05-24c8b6bedea5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" podUID="5b7c43be-2b76-4027-8a05-24c8b6bedea5" Nov 02 20:45:17.378201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.377318 4971 generic.go:356] "Generic (PLEG): container finished" podID="c5cb9679-123e-4d5b-9452-82ae0fac9fe7" containerID="2a39f079aae82d9d21b7f453768abcca8d1411f6af9a7fda5716f46f68c3bcdb" exitCode=60 Nov 02 20:45:17.378201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.377425 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" event={"ID":"c5cb9679-123e-4d5b-9452-82ae0fac9fe7","Type":"ContainerDied","Data":"2a39f079aae82d9d21b7f453768abcca8d1411f6af9a7fda5716f46f68c3bcdb"} Nov 02 20:45:17.378201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.377780 4971 scope.go:122] "RemoveContainer" containerID="2a39f079aae82d9d21b7f453768abcca8d1411f6af9a7fda5716f46f68c3bcdb" Nov 02 20:45:17.378201 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:17.378072 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr_odh-ai-gateway-infra(c5cb9679-123e-4d5b-9452-82ae0fac9fe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" podUID="c5cb9679-123e-4d5b-9452-82ae0fac9fe7" Nov 02 20:45:17.380322 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.380297 4971 scope.go:122] "RemoveContainer" containerID="cd79aa4d775b0bb4cb7797ba36f24ab36d0303b71650c76f26cf422927fdb908" Nov 02 20:45:17.489149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.485959 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr"] Nov 02 20:45:17.818704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.817625 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv"] Nov 02 20:45:17.821624 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:45:17.821589 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7906ad9c_8297_4513_aaf2_8729d580bfa5.slice/crio-7d04e449bdaa1d91042cecbb678f79680991588a2c29dc669ed5043ea2635acc WatchSource:0}: Error finding container 7d04e449bdaa1d91042cecbb678f79680991588a2c29dc669ed5043ea2635acc: Status 404 returned error can't find the container with id 7d04e449bdaa1d91042cecbb678f79680991588a2c29dc669ed5043ea2635acc Nov 02 20:45:17.848219 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.844843 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:45:17.848219 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.844930 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:45:17.848219 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.844975 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:45:17.854458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.854437 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5"] Nov 02 20:45:17.854555 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.854529 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" Nov 02 20:45:17.859259 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.859238 4971 scope.go:122] "RemoveContainer" containerID="ae4802c871ded2d2a74192fc53f5c8c641d0197831605e40a66a066290a503a8" Nov 02 20:45:17.859480 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.859458 4971 scope.go:122] "RemoveContainer" containerID="c0c91ba6424f5843b068ab1d40265dded9c6b06249de844b3e21f66b4225156e" Nov 02 20:45:17.859918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.859663 4971 scope.go:122] "RemoveContainer" containerID="ea8bac6ac85655347858233a9ea5d541a9954ef35bd33eac8176ea2970d2d8ad" Nov 02 20:45:17.859918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.859869 4971 scope.go:122] "RemoveContainer" containerID="d6854a0edce336c48680ec5c05cd43bc0f267531f8f63046f7b6affc9fe98708" Nov 02 20:45:17.908422 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.908387 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:45:17.909262 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.909239 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10dd59b7-c045-4b1a-a27e-b7034e0b653d" path="/var/lib/kubelet/pods/10dd59b7-c045-4b1a-a27e-b7034e0b653d/volumes" Nov 02 20:45:17.910342 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.910258 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf3f8ce-7c86-4fa5-ac99-bf2532808882" path="/var/lib/kubelet/pods/9bf3f8ce-7c86-4fa5-ac99-bf2532808882/volumes" Nov 02 20:45:17.910794 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.910773 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv"] Nov 02 20:45:17.915536 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.915505 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7"] Nov 02 20:45:17.917247 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.917214 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe71e0b5-fe81-4fff-963a-ea781eaceb14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8719adfc-qv6nv\" (UID: \"fe71e0b5-fe81-4fff-963a-ea781eaceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" Nov 02 20:45:17.917617 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.917589 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe71e0b5-fe81-4fff-963a-ea781eaceb14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8719adfc-qv6nv\" (UID: \"fe71e0b5-fe81-4fff-963a-ea781eaceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" Nov 02 20:45:17.917696 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.917637 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7b65\" (UniqueName: \"kubernetes.io/projected/fe71e0b5-fe81-4fff-963a-ea781eaceb14-kube-api-access-j7b65\") pod \"maas-api-revoke-sub-8719adfc-qv6nv\" (UID: \"fe71e0b5-fe81-4fff-963a-ea781eaceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" Nov 02 20:45:17.954105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.933882 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" Nov 02 20:45:17.970329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.970287 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7"] Nov 02 20:45:17.998094 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:17.998046 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:18.020104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.019347 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe71e0b5-fe81-4fff-963a-ea781eaceb14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8719adfc-qv6nv\" (UID: \"fe71e0b5-fe81-4fff-963a-ea781eaceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" Nov 02 20:45:18.020104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.019396 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7b65\" (UniqueName: \"kubernetes.io/projected/fe71e0b5-fe81-4fff-963a-ea781eaceb14-kube-api-access-j7b65\") pod \"maas-api-revoke-sub-8719adfc-qv6nv\" (UID: \"fe71e0b5-fe81-4fff-963a-ea781eaceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" Nov 02 20:45:18.020104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.019432 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63eaa68e-23a6-4d8d-ad62-5e795706b24d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-434192b6-qknx7\" (UID: \"63eaa68e-23a6-4d8d-ad62-5e795706b24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" Nov 02 20:45:18.020104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.019572 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe71e0b5-fe81-4fff-963a-ea781eaceb14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8719adfc-qv6nv\" (UID: \"fe71e0b5-fe81-4fff-963a-ea781eaceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" Nov 02 20:45:18.020104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.019725 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63eaa68e-23a6-4d8d-ad62-5e795706b24d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-434192b6-qknx7\" (UID: \"63eaa68e-23a6-4d8d-ad62-5e795706b24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" Nov 02 20:45:18.020104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.019757 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhtjq\" (UniqueName: \"kubernetes.io/projected/63eaa68e-23a6-4d8d-ad62-5e795706b24d-kube-api-access-xhtjq\") pod \"maas-api-revoke-sub-434192b6-qknx7\" (UID: \"63eaa68e-23a6-4d8d-ad62-5e795706b24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" Nov 02 20:45:18.021066 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.021017 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe71e0b5-fe81-4fff-963a-ea781eaceb14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8719adfc-qv6nv\" (UID: \"fe71e0b5-fe81-4fff-963a-ea781eaceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" Nov 02 20:45:18.056415 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.056388 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe71e0b5-fe81-4fff-963a-ea781eaceb14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8719adfc-qv6nv\" (UID: \"fe71e0b5-fe81-4fff-963a-ea781eaceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" Nov 02 20:45:18.121482 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.121454 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63eaa68e-23a6-4d8d-ad62-5e795706b24d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-434192b6-qknx7\" (UID: \"63eaa68e-23a6-4d8d-ad62-5e795706b24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" Nov 02 20:45:18.121608 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.121493 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhtjq\" (UniqueName: \"kubernetes.io/projected/63eaa68e-23a6-4d8d-ad62-5e795706b24d-kube-api-access-xhtjq\") pod \"maas-api-revoke-sub-434192b6-qknx7\" (UID: \"63eaa68e-23a6-4d8d-ad62-5e795706b24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" Nov 02 20:45:18.121608 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.121552 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63eaa68e-23a6-4d8d-ad62-5e795706b24d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-434192b6-qknx7\" (UID: \"63eaa68e-23a6-4d8d-ad62-5e795706b24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" Nov 02 20:45:18.134184 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.122295 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63eaa68e-23a6-4d8d-ad62-5e795706b24d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-434192b6-qknx7\" (UID: \"63eaa68e-23a6-4d8d-ad62-5e795706b24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" Nov 02 20:45:18.151397 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.147262 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63eaa68e-23a6-4d8d-ad62-5e795706b24d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-434192b6-qknx7\" (UID: \"63eaa68e-23a6-4d8d-ad62-5e795706b24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" Nov 02 20:45:18.152006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.151889 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7b65\" (UniqueName: \"kubernetes.io/projected/fe71e0b5-fe81-4fff-963a-ea781eaceb14-kube-api-access-j7b65\") pod \"maas-api-revoke-sub-8719adfc-qv6nv\" (UID: \"fe71e0b5-fe81-4fff-963a-ea781eaceb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" Nov 02 20:45:18.190923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.190897 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhtjq\" (UniqueName: \"kubernetes.io/projected/63eaa68e-23a6-4d8d-ad62-5e795706b24d-kube-api-access-xhtjq\") pod \"maas-api-revoke-sub-434192b6-qknx7\" (UID: \"63eaa68e-23a6-4d8d-ad62-5e795706b24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" Nov 02 20:45:18.198372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.198349 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" Nov 02 20:45:18.295498 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.295236 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" Nov 02 20:45:18.393980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.392988 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" event={"ID":"7d07f1a3-680b-46aa-9a1a-2f7c0911d411","Type":"ContainerStarted","Data":"3c5454dc67d743df071a2e1b6cb0915797e93236113c8f7dc4b7010ade339ad3"} Nov 02 20:45:18.399529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.398584 4971 generic.go:356] "Generic (PLEG): container finished" podID="37c9b65e-f647-4752-8b1e-ab04fb5257ec" containerID="f326e43b16d3d85be8e4ff592d55b6d178b29a54f3950f7134bec7bc186f0160" exitCode=60 Nov 02 20:45:18.399529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.398686 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" event={"ID":"37c9b65e-f647-4752-8b1e-ab04fb5257ec","Type":"ContainerDied","Data":"f326e43b16d3d85be8e4ff592d55b6d178b29a54f3950f7134bec7bc186f0160"} Nov 02 20:45:18.399529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.398736 4971 scope.go:122] "RemoveContainer" containerID="145e2edfd8b8a6e1e62079fc20240ffefb5ab84a2848a2654a183029224461c7" Nov 02 20:45:18.399529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.399215 4971 scope.go:122] "RemoveContainer" containerID="f326e43b16d3d85be8e4ff592d55b6d178b29a54f3950f7134bec7bc186f0160" Nov 02 20:45:18.400755 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:18.400131 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc_odh-ai-gateway-infra(37c9b65e-f647-4752-8b1e-ab04fb5257ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" podUID="37c9b65e-f647-4752-8b1e-ab04fb5257ec" Nov 02 20:45:18.404677 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.404647 4971 generic.go:356] "Generic (PLEG): container finished" podID="b0bf8d79-25c9-453a-8395-ca13f8f03d16" containerID="ea6e80dd54f63a6adc2d14bc2eee2c86ccaffa5110e6d8cb3f413df6dc624340" exitCode=60 Nov 02 20:45:18.404817 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.404798 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" event={"ID":"b0bf8d79-25c9-453a-8395-ca13f8f03d16","Type":"ContainerDied","Data":"ea6e80dd54f63a6adc2d14bc2eee2c86ccaffa5110e6d8cb3f413df6dc624340"} Nov 02 20:45:18.405226 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.405200 4971 scope.go:122] "RemoveContainer" containerID="ea6e80dd54f63a6adc2d14bc2eee2c86ccaffa5110e6d8cb3f413df6dc624340" Nov 02 20:45:18.405609 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:18.405512 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl_odh-ai-gateway-infra(b0bf8d79-25c9-453a-8395-ca13f8f03d16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" podUID="b0bf8d79-25c9-453a-8395-ca13f8f03d16" Nov 02 20:45:18.431043 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.418732 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" event={"ID":"7906ad9c-8297-4513-aaf2-8729d580bfa5","Type":"ContainerStarted","Data":"78cc9c5d21c26ab1e80a89b299840e5a902490637285aadc77ece1b75fc15b90"} Nov 02 20:45:18.431043 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.418786 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" event={"ID":"7906ad9c-8297-4513-aaf2-8729d580bfa5","Type":"ContainerStarted","Data":"7d04e449bdaa1d91042cecbb678f79680991588a2c29dc669ed5043ea2635acc"} Nov 02 20:45:18.447534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.446799 4971 scope.go:122] "RemoveContainer" containerID="3a865135828834d78368ae307a311752666d4624c1b7208bbd71844a560d943f" Nov 02 20:45:18.490129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.489590 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47"] Nov 02 20:45:18.512552 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.512496 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" podStartSLOduration=18.512476894 podStartE2EDuration="18.512476894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:18.510308529 +0000 UTC m=+1846.799965059" watchObservedRunningTime="2026-11-02 20:45:18.512476894 +0000 UTC m=+1846.802133419" Nov 02 20:45:18.604112 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.601852 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv"] Nov 02 20:45:18.610129 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:45:18.610075 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe71e0b5_fe81_4fff_963a_ea781eaceb14.slice/crio-bad6bbd60094fc6af17602db53a5699fc0c64e60ec4d75c9c08119a62c3e47df WatchSource:0}: Error finding container bad6bbd60094fc6af17602db53a5699fc0c64e60ec4d75c9c08119a62c3e47df: Status 404 returned error can't find the container with id bad6bbd60094fc6af17602db53a5699fc0c64e60ec4d75c9c08119a62c3e47df Nov 02 20:45:18.700114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.700069 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" Nov 02 20:45:18.742574 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.742543 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5cb9679-123e-4d5b-9452-82ae0fac9fe7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5cb9679-123e-4d5b-9452-82ae0fac9fe7-maas-api-service-ca\") pod \"c5cb9679-123e-4d5b-9452-82ae0fac9fe7\" (UID: \"c5cb9679-123e-4d5b-9452-82ae0fac9fe7\") " Nov 02 20:45:18.742855 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.742836 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5cb9679-123e-4d5b-9452-82ae0fac9fe7-kube-api-access-jngjg\" (UniqueName: \"kubernetes.io/projected/c5cb9679-123e-4d5b-9452-82ae0fac9fe7-kube-api-access-jngjg\") pod \"c5cb9679-123e-4d5b-9452-82ae0fac9fe7\" (UID: \"c5cb9679-123e-4d5b-9452-82ae0fac9fe7\") " Nov 02 20:45:18.742916 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.742859 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5cb9679-123e-4d5b-9452-82ae0fac9fe7-maas-api-service-ca" pod "c5cb9679-123e-4d5b-9452-82ae0fac9fe7" (UID: "c5cb9679-123e-4d5b-9452-82ae0fac9fe7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:18.743414 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.743392 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5cb9679-123e-4d5b-9452-82ae0fac9fe7-maas-api-service-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:18.776757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.776732 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7"] Nov 02 20:45:18.824165 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.824052 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5cb9679-123e-4d5b-9452-82ae0fac9fe7-kube-api-access-jngjg" pod "c5cb9679-123e-4d5b-9452-82ae0fac9fe7" (UID: "c5cb9679-123e-4d5b-9452-82ae0fac9fe7"). InnerVolumeSpecName "kube-api-access-jngjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:18.832561 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:45:18.832513 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63eaa68e_23a6_4d8d_ad62_5e795706b24d.slice/crio-06ba5ef67629fd285b666d79d92ccff292d18f0b03592eda8ac283a8626c1e1f WatchSource:0}: Error finding container 06ba5ef67629fd285b666d79d92ccff292d18f0b03592eda8ac283a8626c1e1f: Status 404 returned error can't find the container with id 06ba5ef67629fd285b666d79d92ccff292d18f0b03592eda8ac283a8626c1e1f Nov 02 20:45:18.845442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.845408 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jngjg\" (UniqueName: \"kubernetes.io/projected/c5cb9679-123e-4d5b-9452-82ae0fac9fe7-kube-api-access-jngjg\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:18.849813 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.849781 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:18.853270 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.853242 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" Nov 02 20:45:18.866310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.866264 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" podStartSLOduration=2.866249446 podStartE2EDuration="2.866249446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:18.860450274 +0000 UTC m=+1847.150106802" watchObservedRunningTime="2026-11-02 20:45:18.866249446 +0000 UTC m=+1847.155906042" Nov 02 20:45:18.946108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.946067 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7c43be-2b76-4027-8a05-24c8b6bedea5-kube-api-access-2vc6l\" (UniqueName: \"kubernetes.io/projected/5b7c43be-2b76-4027-8a05-24c8b6bedea5-kube-api-access-2vc6l\") pod \"5b7c43be-2b76-4027-8a05-24c8b6bedea5\" (UID: \"5b7c43be-2b76-4027-8a05-24c8b6bedea5\") " Nov 02 20:45:18.946232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.946160 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b7c43be-2b76-4027-8a05-24c8b6bedea5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7c43be-2b76-4027-8a05-24c8b6bedea5-maas-api-service-ca\") pod \"5b7c43be-2b76-4027-8a05-24c8b6bedea5\" (UID: \"5b7c43be-2b76-4027-8a05-24c8b6bedea5\") " Nov 02 20:45:18.946555 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.946518 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7c43be-2b76-4027-8a05-24c8b6bedea5-maas-api-service-ca" pod "5b7c43be-2b76-4027-8a05-24c8b6bedea5" (UID: "5b7c43be-2b76-4027-8a05-24c8b6bedea5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:18.946854 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.946834 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7c43be-2b76-4027-8a05-24c8b6bedea5-maas-api-service-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:18.948330 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:18.948307 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7c43be-2b76-4027-8a05-24c8b6bedea5-kube-api-access-2vc6l" pod "5b7c43be-2b76-4027-8a05-24c8b6bedea5" (UID: "5b7c43be-2b76-4027-8a05-24c8b6bedea5"). InnerVolumeSpecName "kube-api-access-2vc6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:19.048107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.048065 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vc6l\" (UniqueName: \"kubernetes.io/projected/5b7c43be-2b76-4027-8a05-24c8b6bedea5-kube-api-access-2vc6l\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:19.292777 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.292747 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:19.316124 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.312621 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:19.441849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.441816 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" event={"ID":"5b7c43be-2b76-4027-8a05-24c8b6bedea5","Type":"ContainerDied","Data":"e1f6c9abe9523e464360459a0a56c00f784f99010b2b1fbec27dc596656695b2"} Nov 02 20:45:19.442015 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.441824 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47" Nov 02 20:45:19.442015 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.441902 4971 scope.go:122] "RemoveContainer" containerID="a067d8f0fffd8e026b3bd4855cc9cd440f9e301870d46061b568c9695ae4f252" Nov 02 20:45:19.443994 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.443964 4971 generic.go:356] "Generic (PLEG): container finished" podID="45bc19b5-e425-4cc1-85b1-2b0f25751cfa" containerID="f580747a380c44d60f5949005176fbc4d58a007a9b7ef2bc242fc55cd25d1d41" exitCode=60 Nov 02 20:45:19.444153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.444026 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" event={"ID":"45bc19b5-e425-4cc1-85b1-2b0f25751cfa","Type":"ContainerDied","Data":"f580747a380c44d60f5949005176fbc4d58a007a9b7ef2bc242fc55cd25d1d41"} Nov 02 20:45:19.444506 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.444475 4971 scope.go:122] "RemoveContainer" containerID="f580747a380c44d60f5949005176fbc4d58a007a9b7ef2bc242fc55cd25d1d41" Nov 02 20:45:19.444786 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:19.444760 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x_odh-ai-gateway-infra(45bc19b5-e425-4cc1-85b1-2b0f25751cfa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" podUID="45bc19b5-e425-4cc1-85b1-2b0f25751cfa" Nov 02 20:45:19.446629 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.446607 4971 generic.go:356] "Generic (PLEG): container finished" podID="7d07f1a3-680b-46aa-9a1a-2f7c0911d411" containerID="3c5454dc67d743df071a2e1b6cb0915797e93236113c8f7dc4b7010ade339ad3" exitCode=60 Nov 02 20:45:19.446713 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.446680 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" event={"ID":"7d07f1a3-680b-46aa-9a1a-2f7c0911d411","Type":"ContainerDied","Data":"3c5454dc67d743df071a2e1b6cb0915797e93236113c8f7dc4b7010ade339ad3"} Nov 02 20:45:19.447074 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.447056 4971 scope.go:122] "RemoveContainer" containerID="3c5454dc67d743df071a2e1b6cb0915797e93236113c8f7dc4b7010ade339ad3" Nov 02 20:45:19.447416 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:19.447391 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf_odh-ai-gateway-infra(7d07f1a3-680b-46aa-9a1a-2f7c0911d411)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" podUID="7d07f1a3-680b-46aa-9a1a-2f7c0911d411" Nov 02 20:45:19.449460 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.449444 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" Nov 02 20:45:19.449552 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.449469 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr" event={"ID":"c5cb9679-123e-4d5b-9452-82ae0fac9fe7","Type":"ContainerDied","Data":"3593cb20d0645da98c97517e230f9c7591bf7b1cc28df5e6bb75a916ba9b5634"} Nov 02 20:45:19.455443 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.455425 4971 generic.go:356] "Generic (PLEG): container finished" podID="54901b53-c9d1-43c1-a587-63845bb341cc" containerID="c08ba626bd9b3d0f1bd007d6234b7b42536f98c967a7a77722e2f68a41934885" exitCode=60 Nov 02 20:45:19.455554 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.455500 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" event={"ID":"54901b53-c9d1-43c1-a587-63845bb341cc","Type":"ContainerDied","Data":"c08ba626bd9b3d0f1bd007d6234b7b42536f98c967a7a77722e2f68a41934885"} Nov 02 20:45:19.455928 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.455820 4971 scope.go:122] "RemoveContainer" containerID="c08ba626bd9b3d0f1bd007d6234b7b42536f98c967a7a77722e2f68a41934885" Nov 02 20:45:19.455992 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.455969 4971 scope.go:122] "RemoveContainer" containerID="ea8bac6ac85655347858233a9ea5d541a9954ef35bd33eac8176ea2970d2d8ad" Nov 02 20:45:19.456160 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:19.456135 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c_odh-ai-gateway-infra(54901b53-c9d1-43c1-a587-63845bb341cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" podUID="54901b53-c9d1-43c1-a587-63845bb341cc" Nov 02 20:45:19.457488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.457468 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" event={"ID":"63eaa68e-23a6-4d8d-ad62-5e795706b24d","Type":"ContainerStarted","Data":"21dbc29430303c3cfd7cd2efd396bd078307470a494c77c5b4976a43deb1751c"} Nov 02 20:45:19.457585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.457512 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" event={"ID":"63eaa68e-23a6-4d8d-ad62-5e795706b24d","Type":"ContainerStarted","Data":"06ba5ef67629fd285b666d79d92ccff292d18f0b03592eda8ac283a8626c1e1f"} Nov 02 20:45:19.459177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.459156 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" event={"ID":"fe71e0b5-fe81-4fff-963a-ea781eaceb14","Type":"ContainerStarted","Data":"6cdcb09c1f4650a71fa9fe5eef10e93b1ce30f25446aee37ad32f83c95e994ec"} Nov 02 20:45:19.459275 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.459195 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" event={"ID":"fe71e0b5-fe81-4fff-963a-ea781eaceb14","Type":"ContainerStarted","Data":"bad6bbd60094fc6af17602db53a5699fc0c64e60ec4d75c9c08119a62c3e47df"} Nov 02 20:45:19.461353 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.461329 4971 generic.go:356] "Generic (PLEG): container finished" podID="6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" containerID="337d319af98f3fac06a17bc658c4bfc63ed5d1ece4f4ceb44752ff86b5124930" exitCode=60 Nov 02 20:45:19.461443 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.461370 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" event={"ID":"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5","Type":"ContainerDied","Data":"337d319af98f3fac06a17bc658c4bfc63ed5d1ece4f4ceb44752ff86b5124930"} Nov 02 20:45:19.461841 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.461822 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:19.461982 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.461872 4971 scope.go:122] "RemoveContainer" containerID="337d319af98f3fac06a17bc658c4bfc63ed5d1ece4f4ceb44752ff86b5124930" Nov 02 20:45:19.462436 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:19.462408 4971 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2_odh-ai-gateway-infra(6f52b903-f5e5-4ade-b79e-12c5a1f79ff5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" podUID="6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" Nov 02 20:45:19.462752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.462734 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:45:19.471518 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.471505 4971 scope.go:122] "RemoveContainer" containerID="ae4802c871ded2d2a74192fc53f5c8c641d0197831605e40a66a066290a503a8" Nov 02 20:45:19.486607 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.486591 4971 scope.go:122] "RemoveContainer" containerID="2a39f079aae82d9d21b7f453768abcca8d1411f6af9a7fda5716f46f68c3bcdb" Nov 02 20:45:19.496069 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.496052 4971 scope.go:122] "RemoveContainer" containerID="c0c91ba6424f5843b068ab1d40265dded9c6b06249de844b3e21f66b4225156e" Nov 02 20:45:19.511869 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.511847 4971 scope.go:122] "RemoveContainer" containerID="d6854a0edce336c48680ec5c05cd43bc0f267531f8f63046f7b6affc9fe98708" Nov 02 20:45:19.516995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.516972 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf"] Nov 02 20:45:19.750743 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.750704 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl"] Nov 02 20:45:19.942470 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.942421 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" podStartSLOduration=2.942407552 podStartE2EDuration="2.942407552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:19.939325604 +0000 UTC m=+1848.228982132" watchObservedRunningTime="2026-11-02 20:45:19.942407552 +0000 UTC m=+1848.232064080" Nov 02 20:45:19.983151 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:19.983072 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" Nov 02 20:45:20.062832 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.062789 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bf8d79-25c9-453a-8395-ca13f8f03d16-kube-api-access-pc6xb\" (UniqueName: \"kubernetes.io/projected/b0bf8d79-25c9-453a-8395-ca13f8f03d16-kube-api-access-pc6xb\") pod \"b0bf8d79-25c9-453a-8395-ca13f8f03d16\" (UID: \"b0bf8d79-25c9-453a-8395-ca13f8f03d16\") " Nov 02 20:45:20.063147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.062952 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bf8d79-25c9-453a-8395-ca13f8f03d16-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bf8d79-25c9-453a-8395-ca13f8f03d16-maas-api-service-ca\") pod \"b0bf8d79-25c9-453a-8395-ca13f8f03d16\" (UID: \"b0bf8d79-25c9-453a-8395-ca13f8f03d16\") " Nov 02 20:45:20.063643 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.063606 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bf8d79-25c9-453a-8395-ca13f8f03d16-maas-api-service-ca" pod "b0bf8d79-25c9-453a-8395-ca13f8f03d16" (UID: "b0bf8d79-25c9-453a-8395-ca13f8f03d16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:20.066253 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.066228 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bf8d79-25c9-453a-8395-ca13f8f03d16-kube-api-access-pc6xb" pod "b0bf8d79-25c9-453a-8395-ca13f8f03d16" (UID: "b0bf8d79-25c9-453a-8395-ca13f8f03d16"). InnerVolumeSpecName "kube-api-access-pc6xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:20.093317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.093273 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc"] Nov 02 20:45:20.166104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.165918 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc6xb\" (UniqueName: \"kubernetes.io/projected/b0bf8d79-25c9-453a-8395-ca13f8f03d16-kube-api-access-pc6xb\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:20.167752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.165952 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bf8d79-25c9-453a-8395-ca13f8f03d16-maas-api-service-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:20.169614 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.169554 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47"] Nov 02 20:45:20.176739 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.176717 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-1620dc-29894205-plb47"] Nov 02 20:45:20.267756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.267723 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" Nov 02 20:45:20.369763 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.369720 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37c9b65e-f647-4752-8b1e-ab04fb5257ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37c9b65e-f647-4752-8b1e-ab04fb5257ec-maas-api-service-ca\") pod \"37c9b65e-f647-4752-8b1e-ab04fb5257ec\" (UID: \"37c9b65e-f647-4752-8b1e-ab04fb5257ec\") " Nov 02 20:45:20.370289 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.370049 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c9b65e-f647-4752-8b1e-ab04fb5257ec-kube-api-access-2cj7t\" (UniqueName: \"kubernetes.io/projected/37c9b65e-f647-4752-8b1e-ab04fb5257ec-kube-api-access-2cj7t\") pod \"37c9b65e-f647-4752-8b1e-ab04fb5257ec\" (UID: \"37c9b65e-f647-4752-8b1e-ab04fb5257ec\") " Nov 02 20:45:20.370289 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.370066 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37c9b65e-f647-4752-8b1e-ab04fb5257ec-maas-api-service-ca" pod "37c9b65e-f647-4752-8b1e-ab04fb5257ec" (UID: "37c9b65e-f647-4752-8b1e-ab04fb5257ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:20.370701 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.370680 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37c9b65e-f647-4752-8b1e-ab04fb5257ec-maas-api-service-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:20.372448 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.372422 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c9b65e-f647-4752-8b1e-ab04fb5257ec-kube-api-access-2cj7t" pod "37c9b65e-f647-4752-8b1e-ab04fb5257ec" (UID: "37c9b65e-f647-4752-8b1e-ab04fb5257ec"). InnerVolumeSpecName "kube-api-access-2cj7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:20.471936 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.471894 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cj7t\" (UniqueName: \"kubernetes.io/projected/37c9b65e-f647-4752-8b1e-ab04fb5257ec-kube-api-access-2cj7t\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:20.479998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.479965 4971 generic.go:356] "Generic (PLEG): container finished" podID="fe71e0b5-fe81-4fff-963a-ea781eaceb14" containerID="6cdcb09c1f4650a71fa9fe5eef10e93b1ce30f25446aee37ad32f83c95e994ec" exitCode=0 Nov 02 20:45:20.480181 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.480058 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" event={"ID":"fe71e0b5-fe81-4fff-963a-ea781eaceb14","Type":"ContainerDied","Data":"6cdcb09c1f4650a71fa9fe5eef10e93b1ce30f25446aee37ad32f83c95e994ec"} Nov 02 20:45:20.489635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.489613 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" Nov 02 20:45:20.489635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.489619 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc" event={"ID":"37c9b65e-f647-4752-8b1e-ab04fb5257ec","Type":"ContainerDied","Data":"36dd67d003ab4e784c9b296e0df8656ff5fe7e96f7bf0ba395511798886f6031"} Nov 02 20:45:20.489821 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.489686 4971 scope.go:122] "RemoveContainer" containerID="f326e43b16d3d85be8e4ff592d55b6d178b29a54f3950f7134bec7bc186f0160" Nov 02 20:45:20.491463 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.491441 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" Nov 02 20:45:20.491611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.491526 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl" event={"ID":"b0bf8d79-25c9-453a-8395-ca13f8f03d16","Type":"ContainerDied","Data":"ebe337256aaba98d742c9f218bafb2ce1d4fe48db35c4d2968084cff170e9190"} Nov 02 20:45:20.504999 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.504342 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" podStartSLOduration=3.504323516 podStartE2EDuration="3.504323516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:20.392866548 +0000 UTC m=+1848.682523077" watchObservedRunningTime="2026-11-02 20:45:20.504323516 +0000 UTC m=+1848.793980044" Nov 02 20:45:20.505577 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.505555 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr"] Nov 02 20:45:20.505999 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.505969 4971 scope.go:122] "RemoveContainer" containerID="ea6e80dd54f63a6adc2d14bc2eee2c86ccaffa5110e6d8cb3f413df6dc624340" Nov 02 20:45:20.507051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.506875 4971 generic.go:356] "Generic (PLEG): container finished" podID="63eaa68e-23a6-4d8d-ad62-5e795706b24d" containerID="21dbc29430303c3cfd7cd2efd396bd078307470a494c77c5b4976a43deb1751c" exitCode=0 Nov 02 20:45:20.507051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.506975 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" event={"ID":"63eaa68e-23a6-4d8d-ad62-5e795706b24d","Type":"ContainerDied","Data":"21dbc29430303c3cfd7cd2efd396bd078307470a494c77c5b4976a43deb1751c"} Nov 02 20:45:20.510691 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.510664 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0ea576-29894205-jf4tr"] Nov 02 20:45:20.622406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.622370 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2"] Nov 02 20:45:20.717611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.717579 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" Nov 02 20:45:20.777539 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.777507 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d07f1a3-680b-46aa-9a1a-2f7c0911d411-kube-api-access-znb6q\" (UniqueName: \"kubernetes.io/projected/7d07f1a3-680b-46aa-9a1a-2f7c0911d411-kube-api-access-znb6q\") pod \"7d07f1a3-680b-46aa-9a1a-2f7c0911d411\" (UID: \"7d07f1a3-680b-46aa-9a1a-2f7c0911d411\") " Nov 02 20:45:20.777811 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.777786 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d07f1a3-680b-46aa-9a1a-2f7c0911d411-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d07f1a3-680b-46aa-9a1a-2f7c0911d411-maas-api-service-ca\") pod \"7d07f1a3-680b-46aa-9a1a-2f7c0911d411\" (UID: \"7d07f1a3-680b-46aa-9a1a-2f7c0911d411\") " Nov 02 20:45:20.778254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.778230 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d07f1a3-680b-46aa-9a1a-2f7c0911d411-maas-api-service-ca" pod "7d07f1a3-680b-46aa-9a1a-2f7c0911d411" (UID: "7d07f1a3-680b-46aa-9a1a-2f7c0911d411"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:20.778760 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.778737 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d07f1a3-680b-46aa-9a1a-2f7c0911d411-maas-api-service-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:20.779896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.779873 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d07f1a3-680b-46aa-9a1a-2f7c0911d411-kube-api-access-znb6q" pod "7d07f1a3-680b-46aa-9a1a-2f7c0911d411" (UID: "7d07f1a3-680b-46aa-9a1a-2f7c0911d411"). InnerVolumeSpecName "kube-api-access-znb6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:20.800224 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.800200 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" Nov 02 20:45:20.838251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.838221 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x"] Nov 02 20:45:20.880267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.880219 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f52b903-f5e5-4ade-b79e-12c5a1f79ff5-kube-api-access-gw8gw\" (UniqueName: \"kubernetes.io/projected/6f52b903-f5e5-4ade-b79e-12c5a1f79ff5-kube-api-access-gw8gw\") pod \"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5\" (UID: \"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5\") " Nov 02 20:45:20.880458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.880312 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f52b903-f5e5-4ade-b79e-12c5a1f79ff5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f52b903-f5e5-4ade-b79e-12c5a1f79ff5-maas-api-service-ca\") pod \"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5\" (UID: \"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5\") " Nov 02 20:45:20.880721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.880691 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f52b903-f5e5-4ade-b79e-12c5a1f79ff5-maas-api-service-ca" pod "6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" (UID: "6f52b903-f5e5-4ade-b79e-12c5a1f79ff5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:20.881452 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.881049 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znb6q\" (UniqueName: \"kubernetes.io/projected/7d07f1a3-680b-46aa-9a1a-2f7c0911d411-kube-api-access-znb6q\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:20.881452 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.881075 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f52b903-f5e5-4ade-b79e-12c5a1f79ff5-maas-api-service-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:20.887270 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.887244 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f52b903-f5e5-4ade-b79e-12c5a1f79ff5-kube-api-access-gw8gw" pod "6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" (UID: "6f52b903-f5e5-4ade-b79e-12c5a1f79ff5"). InnerVolumeSpecName "kube-api-access-gw8gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:20.982621 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:20.982588 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw8gw\" (UniqueName: \"kubernetes.io/projected/6f52b903-f5e5-4ade-b79e-12c5a1f79ff5-kube-api-access-gw8gw\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:21.044758 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.044689 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" Nov 02 20:45:21.057354 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.057325 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc"] Nov 02 20:45:21.064804 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.064763 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c49693-29894205-f9ngc"] Nov 02 20:45:21.185606 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.185574 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45bc19b5-e425-4cc1-85b1-2b0f25751cfa-kube-api-access-pg96l\" (UniqueName: \"kubernetes.io/projected/45bc19b5-e425-4cc1-85b1-2b0f25751cfa-kube-api-access-pg96l\") pod \"45bc19b5-e425-4cc1-85b1-2b0f25751cfa\" (UID: \"45bc19b5-e425-4cc1-85b1-2b0f25751cfa\") " Nov 02 20:45:21.185861 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.185835 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45bc19b5-e425-4cc1-85b1-2b0f25751cfa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45bc19b5-e425-4cc1-85b1-2b0f25751cfa-maas-api-service-ca\") pod \"45bc19b5-e425-4cc1-85b1-2b0f25751cfa\" (UID: \"45bc19b5-e425-4cc1-85b1-2b0f25751cfa\") " Nov 02 20:45:21.186255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.186227 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45bc19b5-e425-4cc1-85b1-2b0f25751cfa-maas-api-service-ca" pod "45bc19b5-e425-4cc1-85b1-2b0f25751cfa" (UID: "45bc19b5-e425-4cc1-85b1-2b0f25751cfa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:21.186645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.186623 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45bc19b5-e425-4cc1-85b1-2b0f25751cfa-maas-api-service-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:21.187995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.187969 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bc19b5-e425-4cc1-85b1-2b0f25751cfa-kube-api-access-pg96l" pod "45bc19b5-e425-4cc1-85b1-2b0f25751cfa" (UID: "45bc19b5-e425-4cc1-85b1-2b0f25751cfa"). InnerVolumeSpecName "kube-api-access-pg96l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:21.282907 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.282864 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl"] Nov 02 20:45:21.286940 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.286895 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-61cdc1-29894205-lvfpl"] Nov 02 20:45:21.287936 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.287913 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg96l\" (UniqueName: \"kubernetes.io/projected/45bc19b5-e425-4cc1-85b1-2b0f25751cfa-kube-api-access-pg96l\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:21.294561 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.294534 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c"] Nov 02 20:45:21.507028 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.506992 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" Nov 02 20:45:21.518794 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.518767 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" event={"ID":"54901b53-c9d1-43c1-a587-63845bb341cc","Type":"ContainerDied","Data":"991eb98ebae183396cacd49838da48715214fab607a3da7ce9b2b623c40ea4af"} Nov 02 20:45:21.518939 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.518829 4971 scope.go:122] "RemoveContainer" containerID="c08ba626bd9b3d0f1bd007d6234b7b42536f98c967a7a77722e2f68a41934885" Nov 02 20:45:21.518939 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.518774 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c" Nov 02 20:45:21.520691 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.520673 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" Nov 02 20:45:21.520801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.520699 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2" event={"ID":"6f52b903-f5e5-4ade-b79e-12c5a1f79ff5","Type":"ContainerDied","Data":"cee1f7c2c04342994f5fae6a7b40d551ec897529e7c67c45099139590655e1f5"} Nov 02 20:45:21.522337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.522320 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" Nov 02 20:45:21.522456 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.522365 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x" event={"ID":"45bc19b5-e425-4cc1-85b1-2b0f25751cfa","Type":"ContainerDied","Data":"925947f2f4b95281e9c9d2548596e6b5f2cb8f26e1a5b335b6d3c5aa1fbd9e59"} Nov 02 20:45:21.524390 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.524329 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" event={"ID":"7d07f1a3-680b-46aa-9a1a-2f7c0911d411","Type":"ContainerDied","Data":"6bd64372784920ea2ffa2f19459e948574a97883d72a7ee8ac3c47159f36da91"} Nov 02 20:45:21.524547 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.524521 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf" Nov 02 20:45:21.533504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.533484 4971 scope.go:122] "RemoveContainer" containerID="337d319af98f3fac06a17bc658c4bfc63ed5d1ece4f4ceb44752ff86b5124930" Nov 02 20:45:21.547031 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.547008 4971 scope.go:122] "RemoveContainer" containerID="f580747a380c44d60f5949005176fbc4d58a007a9b7ef2bc242fc55cd25d1d41" Nov 02 20:45:21.564416 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.564386 4971 scope.go:122] "RemoveContainer" containerID="3c5454dc67d743df071a2e1b6cb0915797e93236113c8f7dc4b7010ade339ad3" Nov 02 20:45:21.593132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.592018 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54901b53-c9d1-43c1-a587-63845bb341cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54901b53-c9d1-43c1-a587-63845bb341cc-maas-api-service-ca\") pod \"54901b53-c9d1-43c1-a587-63845bb341cc\" (UID: \"54901b53-c9d1-43c1-a587-63845bb341cc\") " Nov 02 20:45:21.593132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.592209 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54901b53-c9d1-43c1-a587-63845bb341cc-kube-api-access-ptq9q\" (UniqueName: \"kubernetes.io/projected/54901b53-c9d1-43c1-a587-63845bb341cc-kube-api-access-ptq9q\") pod \"54901b53-c9d1-43c1-a587-63845bb341cc\" (UID: \"54901b53-c9d1-43c1-a587-63845bb341cc\") " Nov 02 20:45:21.593132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.592954 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54901b53-c9d1-43c1-a587-63845bb341cc-maas-api-service-ca" pod "54901b53-c9d1-43c1-a587-63845bb341cc" (UID: "54901b53-c9d1-43c1-a587-63845bb341cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:21.610118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.606868 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54901b53-c9d1-43c1-a587-63845bb341cc-kube-api-access-ptq9q" pod "54901b53-c9d1-43c1-a587-63845bb341cc" (UID: "54901b53-c9d1-43c1-a587-63845bb341cc"). InnerVolumeSpecName "kube-api-access-ptq9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:21.631712 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.631638 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" podUID="f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:45:21.694270 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.694121 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptq9q\" (UniqueName: \"kubernetes.io/projected/54901b53-c9d1-43c1-a587-63845bb341cc-kube-api-access-ptq9q\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:21.694270 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.694156 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54901b53-c9d1-43c1-a587-63845bb341cc-maas-api-service-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:21.743293 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.743255 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" Nov 02 20:45:21.752127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.750326 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" Nov 02 20:45:21.761906 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.758745 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:21.789443 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.789414 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x"] Nov 02 20:45:21.793427 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.793404 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2250b4-29894205-b4s6x"] Nov 02 20:45:21.795360 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.795337 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe71e0b5-fe81-4fff-963a-ea781eaceb14-kube-api-access-j7b65\" (UniqueName: \"kubernetes.io/projected/fe71e0b5-fe81-4fff-963a-ea781eaceb14-kube-api-access-j7b65\") pod \"fe71e0b5-fe81-4fff-963a-ea781eaceb14\" (UID: \"fe71e0b5-fe81-4fff-963a-ea781eaceb14\") " Nov 02 20:45:21.795619 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.795528 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe71e0b5-fe81-4fff-963a-ea781eaceb14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe71e0b5-fe81-4fff-963a-ea781eaceb14-maas-api-openshift-ca\") pod \"fe71e0b5-fe81-4fff-963a-ea781eaceb14\" (UID: \"fe71e0b5-fe81-4fff-963a-ea781eaceb14\") " Nov 02 20:45:21.795619 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.795617 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe71e0b5-fe81-4fff-963a-ea781eaceb14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe71e0b5-fe81-4fff-963a-ea781eaceb14-maas-api-xks-ca\") pod \"fe71e0b5-fe81-4fff-963a-ea781eaceb14\" (UID: \"fe71e0b5-fe81-4fff-963a-ea781eaceb14\") " Nov 02 20:45:21.795958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.795927 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe71e0b5-fe81-4fff-963a-ea781eaceb14-maas-api-openshift-ca" pod "fe71e0b5-fe81-4fff-963a-ea781eaceb14" (UID: "fe71e0b5-fe81-4fff-963a-ea781eaceb14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:21.796253 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.796225 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe71e0b5-fe81-4fff-963a-ea781eaceb14-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:21.797879 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.797820 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe71e0b5-fe81-4fff-963a-ea781eaceb14-kube-api-access-j7b65" pod "fe71e0b5-fe81-4fff-963a-ea781eaceb14" (UID: "fe71e0b5-fe81-4fff-963a-ea781eaceb14"). InnerVolumeSpecName "kube-api-access-j7b65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:21.797976 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.797882 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe71e0b5-fe81-4fff-963a-ea781eaceb14-maas-api-xks-ca" pod "fe71e0b5-fe81-4fff-963a-ea781eaceb14" (UID: "fe71e0b5-fe81-4fff-963a-ea781eaceb14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:45:21.870260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.870236 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c9b65e-f647-4752-8b1e-ab04fb5257ec" path="/var/lib/kubelet/pods/37c9b65e-f647-4752-8b1e-ab04fb5257ec/volumes" Nov 02 20:45:21.870644 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.870630 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45bc19b5-e425-4cc1-85b1-2b0f25751cfa" path="/var/lib/kubelet/pods/45bc19b5-e425-4cc1-85b1-2b0f25751cfa/volumes" Nov 02 20:45:21.870993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.870980 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b7c43be-2b76-4027-8a05-24c8b6bedea5" path="/var/lib/kubelet/pods/5b7c43be-2b76-4027-8a05-24c8b6bedea5/volumes" Nov 02 20:45:21.871700 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.871684 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0bf8d79-25c9-453a-8395-ca13f8f03d16" path="/var/lib/kubelet/pods/b0bf8d79-25c9-453a-8395-ca13f8f03d16/volumes" Nov 02 20:45:21.872101 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.872070 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5cb9679-123e-4d5b-9452-82ae0fac9fe7" path="/var/lib/kubelet/pods/c5cb9679-123e-4d5b-9452-82ae0fac9fe7/volumes" Nov 02 20:45:21.896970 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.896936 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63eaa68e-23a6-4d8d-ad62-5e795706b24d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63eaa68e-23a6-4d8d-ad62-5e795706b24d-maas-api-openshift-ca\") pod \"63eaa68e-23a6-4d8d-ad62-5e795706b24d\" (UID: \"63eaa68e-23a6-4d8d-ad62-5e795706b24d\") " Nov 02 20:45:21.897245 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.897226 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63eaa68e-23a6-4d8d-ad62-5e795706b24d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63eaa68e-23a6-4d8d-ad62-5e795706b24d-maas-api-xks-ca\") pod \"63eaa68e-23a6-4d8d-ad62-5e795706b24d\" (UID: \"63eaa68e-23a6-4d8d-ad62-5e795706b24d\") " Nov 02 20:45:21.897310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.897269 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63eaa68e-23a6-4d8d-ad62-5e795706b24d-kube-api-access-xhtjq\" (UniqueName: \"kubernetes.io/projected/63eaa68e-23a6-4d8d-ad62-5e795706b24d-kube-api-access-xhtjq\") pod \"63eaa68e-23a6-4d8d-ad62-5e795706b24d\" (UID: \"63eaa68e-23a6-4d8d-ad62-5e795706b24d\") " Nov 02 20:45:21.897411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.897385 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63eaa68e-23a6-4d8d-ad62-5e795706b24d-maas-api-openshift-ca" pod "63eaa68e-23a6-4d8d-ad62-5e795706b24d" (UID: "63eaa68e-23a6-4d8d-ad62-5e795706b24d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:21.897663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.897643 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe71e0b5-fe81-4fff-963a-ea781eaceb14-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:21.897663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.897664 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7b65\" (UniqueName: \"kubernetes.io/projected/fe71e0b5-fe81-4fff-963a-ea781eaceb14-kube-api-access-j7b65\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:21.897845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.897678 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63eaa68e-23a6-4d8d-ad62-5e795706b24d-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:21.899628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.899602 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63eaa68e-23a6-4d8d-ad62-5e795706b24d-maas-api-xks-ca" pod "63eaa68e-23a6-4d8d-ad62-5e795706b24d" (UID: "63eaa68e-23a6-4d8d-ad62-5e795706b24d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:45:21.899778 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:21.899759 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63eaa68e-23a6-4d8d-ad62-5e795706b24d-kube-api-access-xhtjq" pod "63eaa68e-23a6-4d8d-ad62-5e795706b24d" (UID: "63eaa68e-23a6-4d8d-ad62-5e795706b24d"). InnerVolumeSpecName "kube-api-access-xhtjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:22.000419 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.000100 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63eaa68e-23a6-4d8d-ad62-5e795706b24d-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:22.000419 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.000264 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhtjq\" (UniqueName: \"kubernetes.io/projected/63eaa68e-23a6-4d8d-ad62-5e795706b24d-kube-api-access-xhtjq\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:22.016162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.016136 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2"] Nov 02 20:45:22.022290 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.022267 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-039b89-29894205-76xm2"] Nov 02 20:45:22.039232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.039206 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:45:22.039358 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.039252 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" podUID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:45:22.045736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.045704 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:22.137576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.137544 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf"] Nov 02 20:45:22.140368 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.140343 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4166b0-29894205-nf2nf"] Nov 02 20:45:22.241632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.241602 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c"] Nov 02 20:45:22.245372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.245348 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-654ae4-29894205-78g5c"] Nov 02 20:45:22.536784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.536659 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" Nov 02 20:45:22.537291 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.536667 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7" event={"ID":"63eaa68e-23a6-4d8d-ad62-5e795706b24d","Type":"ContainerDied","Data":"06ba5ef67629fd285b666d79d92ccff292d18f0b03592eda8ac283a8626c1e1f"} Nov 02 20:45:22.537364 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.537330 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06ba5ef67629fd285b666d79d92ccff292d18f0b03592eda8ac283a8626c1e1f" Nov 02 20:45:22.538823 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.538783 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" event={"ID":"fe71e0b5-fe81-4fff-963a-ea781eaceb14","Type":"ContainerDied","Data":"bad6bbd60094fc6af17602db53a5699fc0c64e60ec4d75c9c08119a62c3e47df"} Nov 02 20:45:22.538950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.538839 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bad6bbd60094fc6af17602db53a5699fc0c64e60ec4d75c9c08119a62c3e47df" Nov 02 20:45:22.538950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.538794 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv" Nov 02 20:45:22.636425 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:22.636389 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" podUID="2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:45:23.549762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:23.549727 4971 generic.go:356] "Generic (PLEG): container finished" podID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" containerID="541c576ec9025f95e2e791888e95539786b459018e8def278a153f7ac0008cc5" exitCode=0 Nov 02 20:45:23.550208 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:23.549797 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" event={"ID":"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23","Type":"ContainerDied","Data":"541c576ec9025f95e2e791888e95539786b459018e8def278a153f7ac0008cc5"} Nov 02 20:45:23.550267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:23.550250 4971 scope.go:122] "RemoveContainer" containerID="541c576ec9025f95e2e791888e95539786b459018e8def278a153f7ac0008cc5" Nov 02 20:45:23.551708 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:23.551679 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:45:23.551787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:23.551749 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:45:23.551833 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:23.551784 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:45:23.867289 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:23.867258 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54901b53-c9d1-43c1-a587-63845bb341cc" path="/var/lib/kubelet/pods/54901b53-c9d1-43c1-a587-63845bb341cc/volumes" Nov 02 20:45:23.867722 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:23.867706 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" path="/var/lib/kubelet/pods/6f52b903-f5e5-4ade-b79e-12c5a1f79ff5/volumes" Nov 02 20:45:23.868114 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:23.868095 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d07f1a3-680b-46aa-9a1a-2f7c0911d411" path="/var/lib/kubelet/pods/7d07f1a3-680b-46aa-9a1a-2f7c0911d411/volumes" Nov 02 20:45:24.075863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:24.075828 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:24.559903 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:24.559866 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" event={"ID":"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23","Type":"ContainerStarted","Data":"ae4a11cdd15d312e43afd1071dc944a972e76732dac3279b4a7569afea805341"} Nov 02 20:45:24.798247 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:24.798215 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:45:24.798435 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:24.798417 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 20:45:24.798489 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:24.798466 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" podUID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 20:45:25.798240 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:25.798201 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 20:45:25.798717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:25.798274 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" podUID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 20:45:26.797840 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:26.797810 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 20:45:26.798021 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:26.797858 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" podUID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 20:45:27.239110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.230581 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:27.486124 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.486076 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:27.550573 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.550491 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs"] Nov 02 20:45:27.551129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551112 4971 cpu_manager.go:414] "containerMap: removing container" podUID="45bc19b5-e425-4cc1-85b1-2b0f25751cfa" containerName="cleanup" Nov 02 20:45:27.551185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551132 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45bc19b5-e425-4cc1-85b1-2b0f25751cfa" containerName="cleanup" Nov 02 20:45:27.551185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551142 4971 cpu_manager.go:414] "containerMap: removing container" podUID="54901b53-c9d1-43c1-a587-63845bb341cc" containerName="cleanup" Nov 02 20:45:27.551185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551147 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54901b53-c9d1-43c1-a587-63845bb341cc" containerName="cleanup" Nov 02 20:45:27.551185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551155 4971 cpu_manager.go:414] "containerMap: removing container" podUID="54901b53-c9d1-43c1-a587-63845bb341cc" containerName="cleanup" Nov 02 20:45:27.551185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551160 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54901b53-c9d1-43c1-a587-63845bb341cc" containerName="cleanup" Nov 02 20:45:27.551185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551169 4971 cpu_manager.go:414] "containerMap: removing container" podUID="45bc19b5-e425-4cc1-85b1-2b0f25751cfa" containerName="cleanup" Nov 02 20:45:27.551185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551173 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45bc19b5-e425-4cc1-85b1-2b0f25751cfa" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551192 4971 cpu_manager.go:414] "containerMap: removing container" podUID="5b7c43be-2b76-4027-8a05-24c8b6bedea5" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551197 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7c43be-2b76-4027-8a05-24c8b6bedea5" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551202 4971 cpu_manager.go:414] "containerMap: removing container" podUID="37c9b65e-f647-4752-8b1e-ab04fb5257ec" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551207 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c9b65e-f647-4752-8b1e-ab04fb5257ec" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551212 4971 cpu_manager.go:414] "containerMap: removing container" podUID="b0bf8d79-25c9-453a-8395-ca13f8f03d16" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551216 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bf8d79-25c9-453a-8395-ca13f8f03d16" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551226 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c5cb9679-123e-4d5b-9452-82ae0fac9fe7" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551231 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5cb9679-123e-4d5b-9452-82ae0fac9fe7" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551238 4971 cpu_manager.go:414] "containerMap: removing container" podUID="6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551243 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551249 4971 cpu_manager.go:414] "containerMap: removing container" podUID="b0bf8d79-25c9-453a-8395-ca13f8f03d16" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551253 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bf8d79-25c9-453a-8395-ca13f8f03d16" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551259 4971 cpu_manager.go:414] "containerMap: removing container" podUID="37c9b65e-f647-4752-8b1e-ab04fb5257ec" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551263 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c9b65e-f647-4752-8b1e-ab04fb5257ec" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551267 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c5cb9679-123e-4d5b-9452-82ae0fac9fe7" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551271 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5cb9679-123e-4d5b-9452-82ae0fac9fe7" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551281 4971 cpu_manager.go:414] "containerMap: removing container" podUID="b0bf8d79-25c9-453a-8395-ca13f8f03d16" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551285 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bf8d79-25c9-453a-8395-ca13f8f03d16" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551294 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c5cb9679-123e-4d5b-9452-82ae0fac9fe7" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551298 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5cb9679-123e-4d5b-9452-82ae0fac9fe7" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551303 4971 cpu_manager.go:414] "containerMap: removing container" podUID="6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551306 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551317 4971 cpu_manager.go:414] "containerMap: removing container" podUID="6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551320 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551329 4971 cpu_manager.go:414] "containerMap: removing container" podUID="7d07f1a3-680b-46aa-9a1a-2f7c0911d411" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551332 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d07f1a3-680b-46aa-9a1a-2f7c0911d411" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551343 4971 cpu_manager.go:414] "containerMap: removing container" podUID="63eaa68e-23a6-4d8d-ad62-5e795706b24d" containerName="revoke-keys" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551347 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63eaa68e-23a6-4d8d-ad62-5e795706b24d" containerName="revoke-keys" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551353 4971 cpu_manager.go:414] "containerMap: removing container" podUID="7d07f1a3-680b-46aa-9a1a-2f7c0911d411" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551357 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d07f1a3-680b-46aa-9a1a-2f7c0911d411" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551363 4971 cpu_manager.go:414] "containerMap: removing container" podUID="5b7c43be-2b76-4027-8a05-24c8b6bedea5" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551368 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7c43be-2b76-4027-8a05-24c8b6bedea5" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551379 4971 cpu_manager.go:414] "containerMap: removing container" podUID="7d07f1a3-680b-46aa-9a1a-2f7c0911d411" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551383 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d07f1a3-680b-46aa-9a1a-2f7c0911d411" containerName="cleanup" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551391 4971 cpu_manager.go:414] "containerMap: removing container" podUID="fe71e0b5-fe81-4fff-963a-ea781eaceb14" containerName="revoke-keys" Nov 02 20:45:27.551406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551395 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe71e0b5-fe81-4fff-963a-ea781eaceb14" containerName="revoke-keys" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551481 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="45bc19b5-e425-4cc1-85b1-2b0f25751cfa" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551490 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5cb9679-123e-4d5b-9452-82ae0fac9fe7" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551496 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c9b65e-f647-4752-8b1e-ab04fb5257ec" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551501 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551507 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551512 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d07f1a3-680b-46aa-9a1a-2f7c0911d411" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551517 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="54901b53-c9d1-43c1-a587-63845bb341cc" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551523 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="63eaa68e-23a6-4d8d-ad62-5e795706b24d" containerName="revoke-keys" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551527 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d07f1a3-680b-46aa-9a1a-2f7c0911d411" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551533 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7c43be-2b76-4027-8a05-24c8b6bedea5" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551538 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7c43be-2b76-4027-8a05-24c8b6bedea5" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551542 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d07f1a3-680b-46aa-9a1a-2f7c0911d411" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551548 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe71e0b5-fe81-4fff-963a-ea781eaceb14" containerName="revoke-keys" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551554 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="54901b53-c9d1-43c1-a587-63845bb341cc" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551559 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="45bc19b5-e425-4cc1-85b1-2b0f25751cfa" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551565 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c9b65e-f647-4752-8b1e-ab04fb5257ec" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551569 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bf8d79-25c9-453a-8395-ca13f8f03d16" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551574 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5cb9679-123e-4d5b-9452-82ae0fac9fe7" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551580 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f52b903-f5e5-4ade-b79e-12c5a1f79ff5" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551585 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bf8d79-25c9-453a-8395-ca13f8f03d16" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551590 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c9b65e-f647-4752-8b1e-ab04fb5257ec" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551597 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5cb9679-123e-4d5b-9452-82ae0fac9fe7" containerName="cleanup" Nov 02 20:45:27.552553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.551602 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bf8d79-25c9-453a-8395-ca13f8f03d16" containerName="cleanup" Nov 02 20:45:27.556265 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.556248 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" Nov 02 20:45:27.591292 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.591262 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs"] Nov 02 20:45:27.651106 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.636349 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-58c2e0\"" Nov 02 20:45:27.658105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.651600 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-58c2e0-dockercfg-dckvz\"" Nov 02 20:45:27.747110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.710711 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ddaa0e-4ac9-419b-a751-4351874a0a87-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs\" (UID: \"31ddaa0e-4ac9-419b-a751-4351874a0a87\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" Nov 02 20:45:27.747110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.710826 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6mfl\" (UniqueName: \"kubernetes.io/projected/31ddaa0e-4ac9-419b-a751-4351874a0a87-kube-api-access-b6mfl\") pod \"payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs\" (UID: \"31ddaa0e-4ac9-419b-a751-4351874a0a87\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" Nov 02 20:45:27.812360 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.812254 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 20:45:27.812550 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.812394 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" podUID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 20:45:27.828943 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.815844 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ddaa0e-4ac9-419b-a751-4351874a0a87-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs\" (UID: \"31ddaa0e-4ac9-419b-a751-4351874a0a87\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" Nov 02 20:45:27.828943 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.815985 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6mfl\" (UniqueName: \"kubernetes.io/projected/31ddaa0e-4ac9-419b-a751-4351874a0a87-kube-api-access-b6mfl\") pod \"payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs\" (UID: \"31ddaa0e-4ac9-419b-a751-4351874a0a87\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" Nov 02 20:45:27.828943 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.818679 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ddaa0e-4ac9-419b-a751-4351874a0a87-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs\" (UID: \"31ddaa0e-4ac9-419b-a751-4351874a0a87\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" Nov 02 20:45:27.828943 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.820646 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn"] Nov 02 20:45:27.828943 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.822580 4971 cpu_manager.go:414] "containerMap: removing container" podUID="54901b53-c9d1-43c1-a587-63845bb341cc" containerName="cleanup" Nov 02 20:45:27.828943 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.822602 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54901b53-c9d1-43c1-a587-63845bb341cc" containerName="cleanup" Nov 02 20:45:27.863501 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.863472 4971 cpu_manager.go:414] "containerMap: removing container" podUID="45bc19b5-e425-4cc1-85b1-2b0f25751cfa" containerName="cleanup" Nov 02 20:45:27.863683 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.863672 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45bc19b5-e425-4cc1-85b1-2b0f25751cfa" containerName="cleanup" Nov 02 20:45:27.863775 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.863767 4971 cpu_manager.go:414] "containerMap: removing container" podUID="5b7c43be-2b76-4027-8a05-24c8b6bedea5" containerName="cleanup" Nov 02 20:45:27.863826 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.863819 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7c43be-2b76-4027-8a05-24c8b6bedea5" containerName="cleanup" Nov 02 20:45:27.863890 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.863883 4971 cpu_manager.go:414] "containerMap: removing container" podUID="37c9b65e-f647-4752-8b1e-ab04fb5257ec" containerName="cleanup" Nov 02 20:45:27.863938 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.863932 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c9b65e-f647-4752-8b1e-ab04fb5257ec" containerName="cleanup" Nov 02 20:45:27.864251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.864237 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="54901b53-c9d1-43c1-a587-63845bb341cc" containerName="cleanup" Nov 02 20:45:27.864352 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.864344 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7c43be-2b76-4027-8a05-24c8b6bedea5" containerName="cleanup" Nov 02 20:45:27.864409 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.864403 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="45bc19b5-e425-4cc1-85b1-2b0f25751cfa" containerName="cleanup" Nov 02 20:45:27.870858 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.870834 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" Nov 02 20:45:27.879655 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.879633 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn"] Nov 02 20:45:27.883879 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.883857 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6mfl\" (UniqueName: \"kubernetes.io/projected/31ddaa0e-4ac9-419b-a751-4351874a0a87-kube-api-access-b6mfl\") pod \"payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs\" (UID: \"31ddaa0e-4ac9-419b-a751-4351874a0a87\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" Nov 02 20:45:27.917760 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.917731 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4e9d3b-0707-402c-b5ae-7d09880c4f18-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn\" (UID: \"2e4e9d3b-0707-402c-b5ae-7d09880c4f18\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" Nov 02 20:45:27.917855 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:27.917779 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f2cv\" (UniqueName: \"kubernetes.io/projected/2e4e9d3b-0707-402c-b5ae-7d09880c4f18-kube-api-access-2f2cv\") pod \"payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn\" (UID: \"2e4e9d3b-0707-402c-b5ae-7d09880c4f18\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" Nov 02 20:45:28.018758 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:28.018727 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4e9d3b-0707-402c-b5ae-7d09880c4f18-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn\" (UID: \"2e4e9d3b-0707-402c-b5ae-7d09880c4f18\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" Nov 02 20:45:28.018920 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:28.018769 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f2cv\" (UniqueName: \"kubernetes.io/projected/2e4e9d3b-0707-402c-b5ae-7d09880c4f18-kube-api-access-2f2cv\") pod \"payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn\" (UID: \"2e4e9d3b-0707-402c-b5ae-7d09880c4f18\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" Nov 02 20:45:28.019510 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:28.019489 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4e9d3b-0707-402c-b5ae-7d09880c4f18-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn\" (UID: \"2e4e9d3b-0707-402c-b5ae-7d09880c4f18\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" Nov 02 20:45:28.076717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:28.076649 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f2cv\" (UniqueName: \"kubernetes.io/projected/2e4e9d3b-0707-402c-b5ae-7d09880c4f18-kube-api-access-2f2cv\") pod \"payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn\" (UID: \"2e4e9d3b-0707-402c-b5ae-7d09880c4f18\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" Nov 02 20:45:28.170048 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:28.170018 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" Nov 02 20:45:28.201866 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:28.201834 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" Nov 02 20:45:28.573959 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:45:28.573928 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ddaa0e_4ac9_419b_a751_4351874a0a87.slice/crio-179b3f9963e94120934b5cb3bb1de731bf79165455950d225664c60ee3ee727c WatchSource:0}: Error finding container 179b3f9963e94120934b5cb3bb1de731bf79165455950d225664c60ee3ee727c: Status 404 returned error can't find the container with id 179b3f9963e94120934b5cb3bb1de731bf79165455950d225664c60ee3ee727c Nov 02 20:45:28.575139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:28.575113 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs"] Nov 02 20:45:28.602680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:28.602652 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" event={"ID":"31ddaa0e-4ac9-419b-a751-4351874a0a87","Type":"ContainerStarted","Data":"179b3f9963e94120934b5cb3bb1de731bf79165455950d225664c60ee3ee727c"} Nov 02 20:45:28.644216 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:28.622206 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn"] Nov 02 20:45:28.648352 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:45:28.648321 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4e9d3b_0707_402c_b5ae_7d09880c4f18.slice/crio-c90ef8c8dffe59412dfab3b3baa934bc7f9b40ba4ce13ade8071ad6459000957 WatchSource:0}: Error finding container c90ef8c8dffe59412dfab3b3baa934bc7f9b40ba4ce13ade8071ad6459000957: Status 404 returned error can't find the container with id c90ef8c8dffe59412dfab3b3baa934bc7f9b40ba4ce13ade8071ad6459000957 Nov 02 20:45:28.798025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:28.797994 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 02 20:45:28.798244 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:28.798050 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" podUID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 02 20:45:29.415415 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:29.415381 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:29.611914 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:29.611882 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" event={"ID":"31ddaa0e-4ac9-419b-a751-4351874a0a87","Type":"ContainerStarted","Data":"b82eb6b49080ec54e0cf9466eea230aa90946ec8c96c843146a44c3f5509bf0b"} Nov 02 20:45:29.612427 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:29.611977 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" Nov 02 20:45:29.613803 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:29.613777 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" event={"ID":"2e4e9d3b-0707-402c-b5ae-7d09880c4f18","Type":"ContainerStarted","Data":"1b8681c41eda5cbd0e450e42cc6fde6fc6ee63123c51c9b7e8ccb7ddb980ab42"} Nov 02 20:45:29.613911 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:29.613820 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" event={"ID":"2e4e9d3b-0707-402c-b5ae-7d09880c4f18","Type":"ContainerStarted","Data":"c90ef8c8dffe59412dfab3b3baa934bc7f9b40ba4ce13ade8071ad6459000957"} Nov 02 20:45:29.613911 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:29.613848 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" Nov 02 20:45:29.673398 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:29.673301 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:29.721934 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:29.721874 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" podStartSLOduration=2.72185851 podStartE2EDuration="2.72185851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:29.720966359 +0000 UTC m=+1858.010622888" watchObservedRunningTime="2026-11-02 20:45:29.72185851 +0000 UTC m=+1858.011515039" Nov 02 20:45:29.834739 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:29.834680 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" podStartSLOduration=2.834661772 podStartE2EDuration="2.834661772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:29.832875859 +0000 UTC m=+1858.122532391" watchObservedRunningTime="2026-11-02 20:45:29.834661772 +0000 UTC m=+1858.124318299" Nov 02 20:45:29.917660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:29.917629 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:45:29.918413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:29.918380 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:45:29.919583 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:29.919561 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:45:29.927865 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:29.927790 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:30.232366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.232285 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p"] Nov 02 20:45:30.236845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.236826 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.296390 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.296360 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p"] Nov 02 20:45:30.363856 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.363829 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-64ff15-openshift-default-dockercfg-tjm8j\"" Nov 02 20:45:30.407407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.394361 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-workload-certs\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.407407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.394494 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-workload-socket\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.407407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.394533 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-credential-socket\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.407407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.394710 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-envoy\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.407407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.394752 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nxg4\" (UniqueName: \"kubernetes.io/projected/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-kube-api-access-9nxg4\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.407407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.394781 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-data\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.407407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.394810 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-token\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.407407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.394846 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-podinfo\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.407407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.394921 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istiod-ca-cert\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.460113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.459624 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:30.505131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.501628 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-envoy\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.505131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.501686 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nxg4\" (UniqueName: \"kubernetes.io/projected/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-kube-api-access-9nxg4\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.505131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.502066 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-data\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.505131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.502115 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-token\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.505131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.502163 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-podinfo\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.505131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.502255 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istiod-ca-cert\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.505131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.502336 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-workload-certs\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.505131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.502453 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-workload-socket\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.505131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.502483 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-credential-socket\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.505131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.502945 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-credential-socket\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.505131 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.503219 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-data\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.509109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.506563 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-workload-certs\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.509109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.507397 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istiod-ca-cert\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.509109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.507645 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-workload-socket\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.528752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.521805 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-podinfo\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.528752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.528735 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-envoy\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.560624 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.560591 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-token\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.560782 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.560647 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nxg4\" (UniqueName: \"kubernetes.io/projected/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-kube-api-access-9nxg4\") pod \"e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.853020 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.852990 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:30.882679 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:30.882649 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:31.276240 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:31.276212 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p"] Nov 02 20:45:31.276552 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:45:31.276526 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab9301b_293b_420e_8fb7_ad6eeb6acaa3.slice/crio-953441e0c60afe53901133ee72e81b77467bd742a73239d3cd5ad180d13b1cfa WatchSource:0}: Error finding container 953441e0c60afe53901133ee72e81b77467bd742a73239d3cd5ad180d13b1cfa: Status 404 returned error can't find the container with id 953441e0c60afe53901133ee72e81b77467bd742a73239d3cd5ad180d13b1cfa Nov 02 20:45:31.279398 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:31.279365 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:45:31.279473 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:31.279430 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:45:31.279473 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:31.279460 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:45:31.631167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:31.631128 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" podUID="f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:45:31.632029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:31.631996 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" event={"ID":"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3","Type":"ContainerStarted","Data":"9833fdf1e58051ac04e9ebac069cb4f3052cb38a7762058a31623d9bc1c83c1f"} Nov 02 20:45:31.632197 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:31.632059 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" event={"ID":"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3","Type":"ContainerStarted","Data":"953441e0c60afe53901133ee72e81b77467bd742a73239d3cd5ad180d13b1cfa"} Nov 02 20:45:31.746050 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:31.745996 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" podStartSLOduration=1.7459788299999999 podStartE2EDuration="1.74597883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:31.745235681 +0000 UTC m=+1860.034892221" watchObservedRunningTime="2026-11-02 20:45:31.74597883 +0000 UTC m=+1860.035635358" Nov 02 20:45:31.853204 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:31.853175 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:32.420135 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.415992 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:32.633418 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.633202 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" Nov 02 20:45:32.650372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.650343 4971 generic.go:356] "Generic (PLEG): container finished" podID="f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c" containerID="07b7d016cb65a3c2559cd8bcee7e975aa8da695985f1a5ee49aae19d7d18beb3" exitCode=0 Nov 02 20:45:32.650530 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.650503 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" event={"ID":"f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c","Type":"ContainerDied","Data":"07b7d016cb65a3c2559cd8bcee7e975aa8da695985f1a5ee49aae19d7d18beb3"} Nov 02 20:45:32.652391 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.652361 4971 generic.go:356] "Generic (PLEG): container finished" podID="2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3" containerID="329c36247d67f0b919b5878124ef7f872924c5ff9564961b3f0b0931a3818a43" exitCode=0 Nov 02 20:45:32.652530 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.652477 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" Nov 02 20:45:32.652601 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.652525 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" event={"ID":"2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3","Type":"ContainerDied","Data":"329c36247d67f0b919b5878124ef7f872924c5ff9564961b3f0b0931a3818a43"} Nov 02 20:45:32.652601 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.652587 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v" event={"ID":"2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3","Type":"ContainerDied","Data":"1951a294063694d07b2d1b51ab4c516f4c944d3233f11eea3bd524d4ef178e34"} Nov 02 20:45:32.652696 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.652621 4971 scope.go:122] "RemoveContainer" containerID="329c36247d67f0b919b5878124ef7f872924c5ff9564961b3f0b0931a3818a43" Nov 02 20:45:32.665445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.665425 4971 scope.go:122] "RemoveContainer" containerID="329c36247d67f0b919b5878124ef7f872924c5ff9564961b3f0b0931a3818a43" Nov 02 20:45:32.665769 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:32.665744 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"329c36247d67f0b919b5878124ef7f872924c5ff9564961b3f0b0931a3818a43\": container with ID starting with 329c36247d67f0b919b5878124ef7f872924c5ff9564961b3f0b0931a3818a43 not found: ID does not exist" containerID="329c36247d67f0b919b5878124ef7f872924c5ff9564961b3f0b0931a3818a43" Nov 02 20:45:32.665884 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.665779 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"329c36247d67f0b919b5878124ef7f872924c5ff9564961b3f0b0931a3818a43"} err="failed to get container status \"329c36247d67f0b919b5878124ef7f872924c5ff9564961b3f0b0931a3818a43\": rpc error: code = NotFound desc = could not find container \"329c36247d67f0b919b5878124ef7f872924c5ff9564961b3f0b0931a3818a43\": container with ID starting with 329c36247d67f0b919b5878124ef7f872924c5ff9564961b3f0b0931a3818a43 not found: ID does not exist" Nov 02 20:45:32.712515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.712495 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" Nov 02 20:45:32.735197 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.735170 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3-kube-api-access-nv2ft\" (UniqueName: \"kubernetes.io/projected/2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3-kube-api-access-nv2ft\") pod \"2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3\" (UID: \"2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3\") " Nov 02 20:45:32.735297 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.735214 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3-plugins-config-volume\") pod \"2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3\" (UID: \"2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3\") " Nov 02 20:45:32.735635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.735608 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3-plugins-config-volume" pod "2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3" (UID: "2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:32.735756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.735729 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:32.737509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.737490 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3-kube-api-access-nv2ft" pod "2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3" (UID: "2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3"). InnerVolumeSpecName "kube-api-access-nv2ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:32.836955 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.836886 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c-kube-api-access-7mc2v\" (UniqueName: \"kubernetes.io/projected/f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c-kube-api-access-7mc2v\") pod \"f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c\" (UID: \"f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c\") " Nov 02 20:45:32.837147 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.837013 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c-plugins-config-volume\") pod \"f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c\" (UID: \"f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c\") " Nov 02 20:45:32.837388 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.837363 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c-plugins-config-volume" pod "f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c" (UID: "f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:32.837692 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.837639 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv2ft\" (UniqueName: \"kubernetes.io/projected/2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3-kube-api-access-nv2ft\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:32.837692 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.837657 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:32.839578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.839556 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c-kube-api-access-7mc2v" pod "f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c" (UID: "f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c"). InnerVolumeSpecName "kube-api-access-7mc2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:32.862824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.862803 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:32.938811 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:32.938784 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mc2v\" (UniqueName: \"kubernetes.io/projected/f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c-kube-api-access-7mc2v\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:33.120942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:33.120914 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v"] Nov 02 20:45:33.126916 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:33.126888 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bce768-6fb95f8c8d-s7w8v"] Nov 02 20:45:33.340604 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:45:33.340568 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:45:33.661846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:33.661817 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" Nov 02 20:45:33.661846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:33.661837 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j" event={"ID":"f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c","Type":"ContainerDied","Data":"adb961d880f859e4097b968508da879d9e4a4a7ed556cae240093d85ae44137a"} Nov 02 20:45:33.662120 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:33.661931 4971 scope.go:122] "RemoveContainer" containerID="07b7d016cb65a3c2559cd8bcee7e975aa8da695985f1a5ee49aae19d7d18beb3" Nov 02 20:45:33.663413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:33.663390 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:33.664502 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:33.664477 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:45:33.818528 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:33.818458 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j"] Nov 02 20:45:33.824274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:33.824250 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bce768-59755ddb55-pxt9j"] Nov 02 20:45:33.867796 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:33.867761 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3" path="/var/lib/kubelet/pods/2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3/volumes" Nov 02 20:45:33.868214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:33.868194 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c" path="/var/lib/kubelet/pods/f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c/volumes" Nov 02 20:45:34.944698 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:34.944664 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:35.363100 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:35.363026 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:35.613469 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:35.613390 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:38.345513 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:38.345477 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:38.587530 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:38.587484 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:38.832452 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:38.832419 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:39.457562 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.457525 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:39.487122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.480778 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg"] Nov 02 20:45:39.487122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.484556 4971 cpu_manager.go:414] "containerMap: removing container" podUID="2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3" containerName="payload-processing" Nov 02 20:45:39.487122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.484583 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3" containerName="payload-processing" Nov 02 20:45:39.487122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.484628 4971 cpu_manager.go:414] "containerMap: removing container" podUID="f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c" containerName="payload-pre-processing" Nov 02 20:45:39.487122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.484637 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c" containerName="payload-pre-processing" Nov 02 20:45:39.487122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.484948 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3075dbf-1cf7-4f40-9b8a-a8d6423ed53c" containerName="payload-pre-processing" Nov 02 20:45:39.487122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.484990 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ef8c062-bd4c-4aaa-b95a-d56c1c7c68b3" containerName="payload-processing" Nov 02 20:45:39.500461 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.500426 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg" Nov 02 20:45:39.519730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.519133 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v"] Nov 02 20:45:39.519730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.519380 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" podUID="7a943a64-646e-4533-a37c-71334481ec3a" containerName="istio-proxy" containerID="cri-o://199e900c640bece93cdb58672d49d1d38d0b04f0b5ae6c965d3f41e972f210f9" gracePeriod=30 Nov 02 20:45:39.530988 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.529470 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg"] Nov 02 20:45:39.539479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.538907 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx"] Nov 02 20:45:39.544824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.544795 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx" Nov 02 20:45:39.584961 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.584678 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:45:39.584961 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.584828 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:45:39.623192 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.623158 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx"] Nov 02 20:45:39.626567 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.626542 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866d5234-n6rcg\" (UID: \"42bde7ba-8b52-46dc-98a4-ae6df715ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg" Nov 02 20:45:39.626726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.626573 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76qr4\" (UniqueName: \"kubernetes.io/projected/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-kube-api-access-76qr4\") pod \"maas-api-revoke-sub-866d5234-n6rcg\" (UID: \"42bde7ba-8b52-46dc-98a4-ae6df715ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg" Nov 02 20:45:39.626795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.626724 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866d5234-n6rcg\" (UID: \"42bde7ba-8b52-46dc-98a4-ae6df715ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg" Nov 02 20:45:39.714871 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.714762 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:39.728028 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.727989 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866d5234-n6rcg\" (UID: \"42bde7ba-8b52-46dc-98a4-ae6df715ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg" Nov 02 20:45:39.728259 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.728040 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76qr4\" (UniqueName: \"kubernetes.io/projected/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-kube-api-access-76qr4\") pod \"maas-api-revoke-sub-866d5234-n6rcg\" (UID: \"42bde7ba-8b52-46dc-98a4-ae6df715ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg" Nov 02 20:45:39.728259 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.728231 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866d5234-n6rcg\" (UID: \"42bde7ba-8b52-46dc-98a4-ae6df715ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg" Nov 02 20:45:39.728432 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.728405 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5eee46-301a-46eb-8672-458a4b62a3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4a2cb3d-kg8vx\" (UID: \"3b5eee46-301a-46eb-8672-458a4b62a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx" Nov 02 20:45:39.728570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.728468 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8x5l\" (UniqueName: \"kubernetes.io/projected/3b5eee46-301a-46eb-8672-458a4b62a3ec-kube-api-access-z8x5l\") pod \"maas-api-revoke-sub-d4a2cb3d-kg8vx\" (UID: \"3b5eee46-301a-46eb-8672-458a4b62a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx" Nov 02 20:45:39.728570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.728514 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5eee46-301a-46eb-8672-458a4b62a3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4a2cb3d-kg8vx\" (UID: \"3b5eee46-301a-46eb-8672-458a4b62a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx" Nov 02 20:45:39.729438 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.729412 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-866d5234-n6rcg\" (UID: \"42bde7ba-8b52-46dc-98a4-ae6df715ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg" Nov 02 20:45:39.733246 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.733175 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-866d5234-n6rcg\" (UID: \"42bde7ba-8b52-46dc-98a4-ae6df715ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg" Nov 02 20:45:39.773562 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.773525 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf"] Nov 02 20:45:39.779543 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.779517 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" Nov 02 20:45:39.788130 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.788102 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76qr4\" (UniqueName: \"kubernetes.io/projected/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-kube-api-access-76qr4\") pod \"maas-api-revoke-sub-866d5234-n6rcg\" (UID: \"42bde7ba-8b52-46dc-98a4-ae6df715ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg" Nov 02 20:45:39.824136 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.824108 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf"] Nov 02 20:45:39.829284 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.829249 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-64ff15-dockercfg-6d7m7\"" Nov 02 20:45:39.829393 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.829307 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-64ff15\"" Nov 02 20:45:39.829669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.829644 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5eee46-301a-46eb-8672-458a4b62a3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4a2cb3d-kg8vx\" (UID: \"3b5eee46-301a-46eb-8672-458a4b62a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx" Nov 02 20:45:39.829978 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.829957 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5eee46-301a-46eb-8672-458a4b62a3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4a2cb3d-kg8vx\" (UID: \"3b5eee46-301a-46eb-8672-458a4b62a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx" Nov 02 20:45:39.830030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.830016 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8x5l\" (UniqueName: \"kubernetes.io/projected/3b5eee46-301a-46eb-8672-458a4b62a3ec-kube-api-access-z8x5l\") pod \"maas-api-revoke-sub-d4a2cb3d-kg8vx\" (UID: \"3b5eee46-301a-46eb-8672-458a4b62a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx" Nov 02 20:45:39.831029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.830999 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5eee46-301a-46eb-8672-458a4b62a3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4a2cb3d-kg8vx\" (UID: \"3b5eee46-301a-46eb-8672-458a4b62a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx" Nov 02 20:45:39.831188 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.831153 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg" Nov 02 20:45:39.832799 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.832780 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5eee46-301a-46eb-8672-458a4b62a3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4a2cb3d-kg8vx\" (UID: \"3b5eee46-301a-46eb-8672-458a4b62a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx" Nov 02 20:45:39.941119 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.935666 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8x5l\" (UniqueName: \"kubernetes.io/projected/3b5eee46-301a-46eb-8672-458a4b62a3ec-kube-api-access-z8x5l\") pod \"maas-api-revoke-sub-d4a2cb3d-kg8vx\" (UID: \"3b5eee46-301a-46eb-8672-458a4b62a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx" Nov 02 20:45:39.941119 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.940674 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06a7d33d-32b2-4efe-8373-40491c26931d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf\" (UID: \"06a7d33d-32b2-4efe-8373-40491c26931d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" Nov 02 20:45:39.941119 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.940840 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrbdz\" (UniqueName: \"kubernetes.io/projected/06a7d33d-32b2-4efe-8373-40491c26931d-kube-api-access-vrbdz\") pod \"payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf\" (UID: \"06a7d33d-32b2-4efe-8373-40491c26931d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" Nov 02 20:45:39.952127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:39.944928 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx" Nov 02 20:45:40.008863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.008170 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8"] Nov 02 20:45:40.048947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.048910 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8"] Nov 02 20:45:40.049173 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.049038 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" Nov 02 20:45:40.049292 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.049267 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06a7d33d-32b2-4efe-8373-40491c26931d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf\" (UID: \"06a7d33d-32b2-4efe-8373-40491c26931d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" Nov 02 20:45:40.049599 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.049579 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrbdz\" (UniqueName: \"kubernetes.io/projected/06a7d33d-32b2-4efe-8373-40491c26931d-kube-api-access-vrbdz\") pod \"payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf\" (UID: \"06a7d33d-32b2-4efe-8373-40491c26931d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" Nov 02 20:45:40.050827 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.050726 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06a7d33d-32b2-4efe-8373-40491c26931d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf\" (UID: \"06a7d33d-32b2-4efe-8373-40491c26931d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" Nov 02 20:45:40.118219 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.116298 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrbdz\" (UniqueName: \"kubernetes.io/projected/06a7d33d-32b2-4efe-8373-40491c26931d-kube-api-access-vrbdz\") pod \"payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf\" (UID: \"06a7d33d-32b2-4efe-8373-40491c26931d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" Nov 02 20:45:40.150736 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.150708 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9525e1da-30ce-46db-8588-8f67c46da099-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8\" (UID: \"9525e1da-30ce-46db-8588-8f67c46da099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" Nov 02 20:45:40.150958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.150942 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n254d\" (UniqueName: \"kubernetes.io/projected/9525e1da-30ce-46db-8588-8f67c46da099-kube-api-access-n254d\") pod \"payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8\" (UID: \"9525e1da-30ce-46db-8588-8f67c46da099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" Nov 02 20:45:40.252017 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.251985 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n254d\" (UniqueName: \"kubernetes.io/projected/9525e1da-30ce-46db-8588-8f67c46da099-kube-api-access-n254d\") pod \"payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8\" (UID: \"9525e1da-30ce-46db-8588-8f67c46da099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" Nov 02 20:45:40.252241 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.252104 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9525e1da-30ce-46db-8588-8f67c46da099-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8\" (UID: \"9525e1da-30ce-46db-8588-8f67c46da099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" Nov 02 20:45:40.252754 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.252731 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9525e1da-30ce-46db-8588-8f67c46da099-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8\" (UID: \"9525e1da-30ce-46db-8588-8f67c46da099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" Nov 02 20:45:40.310514 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.310446 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n254d\" (UniqueName: \"kubernetes.io/projected/9525e1da-30ce-46db-8588-8f67c46da099-kube-api-access-n254d\") pod \"payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8\" (UID: \"9525e1da-30ce-46db-8588-8f67c46da099\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" Nov 02 20:45:40.373345 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.373317 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" Nov 02 20:45:40.399646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.399620 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg"] Nov 02 20:45:40.399826 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:45:40.399802 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42bde7ba_8b52_46dc_98a4_ae6df715ceb0.slice/crio-ae6faca71170c8840f9c8a064049398caba6a8e45156011edb306160f53a8d10 WatchSource:0}: Error finding container ae6faca71170c8840f9c8a064049398caba6a8e45156011edb306160f53a8d10: Status 404 returned error can't find the container with id ae6faca71170c8840f9c8a064049398caba6a8e45156011edb306160f53a8d10 Nov 02 20:45:40.409267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.409242 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" Nov 02 20:45:40.453457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.450025 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx"] Nov 02 20:45:40.628379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.628351 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" Nov 02 20:45:40.628819 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.628437 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" Nov 02 20:45:40.724647 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.724578 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg" event={"ID":"42bde7ba-8b52-46dc-98a4-ae6df715ceb0","Type":"ContainerStarted","Data":"ae6faca71170c8840f9c8a064049398caba6a8e45156011edb306160f53a8d10"} Nov 02 20:45:40.726582 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.726556 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx" event={"ID":"3b5eee46-301a-46eb-8672-458a4b62a3ec","Type":"ContainerStarted","Data":"7a054e6cadb85b63d236d2c7081cb1822f224912efca241da0f50c280760468b"} Nov 02 20:45:40.756488 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:45:40.756454 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9525e1da_30ce_46db_8588_8f67c46da099.slice/crio-01368f0211d8d4e272f21c545d4e6ef28cc5ad491fc7cdc403ab33ed99325ac8 WatchSource:0}: Error finding container 01368f0211d8d4e272f21c545d4e6ef28cc5ad491fc7cdc403ab33ed99325ac8: Status 404 returned error can't find the container with id 01368f0211d8d4e272f21c545d4e6ef28cc5ad491fc7cdc403ab33ed99325ac8 Nov 02 20:45:40.758670 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.758608 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:40.758833 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.758803 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8"] Nov 02 20:45:40.810116 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:40.810072 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf"] Nov 02 20:45:41.735844 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:41.735767 4971 generic.go:356] "Generic (PLEG): container finished" podID="3b5eee46-301a-46eb-8672-458a4b62a3ec" containerID="851ca3015667062d387db0ab81c090498df8f46930f6ab83cffb1cf0e5a24f75" exitCode=0 Nov 02 20:45:41.736256 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:41.735852 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx" event={"ID":"3b5eee46-301a-46eb-8672-458a4b62a3ec","Type":"ContainerDied","Data":"851ca3015667062d387db0ab81c090498df8f46930f6ab83cffb1cf0e5a24f75"} Nov 02 20:45:41.737752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:41.737724 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" event={"ID":"06a7d33d-32b2-4efe-8373-40491c26931d","Type":"ContainerStarted","Data":"c028c544307efc9c3e74c5c5167e9dab37a5a645013f8e67a2fc39856f1ca5ec"} Nov 02 20:45:41.737900 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:41.737770 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" event={"ID":"06a7d33d-32b2-4efe-8373-40491c26931d","Type":"ContainerStarted","Data":"1cb152ba13ae7d306a6d31b2ebff6a439c78f1ad74a305716d2dd86df08606cf"} Nov 02 20:45:41.737900 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:41.737795 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" Nov 02 20:45:41.744502 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:41.744469 4971 generic.go:356] "Generic (PLEG): container finished" podID="42bde7ba-8b52-46dc-98a4-ae6df715ceb0" containerID="a5845748fb4ef9642240cd69a7f37dc4688643dc386523da250dc602ba0124dd" exitCode=0 Nov 02 20:45:41.744632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:41.744572 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg" event={"ID":"42bde7ba-8b52-46dc-98a4-ae6df715ceb0","Type":"ContainerDied","Data":"a5845748fb4ef9642240cd69a7f37dc4688643dc386523da250dc602ba0124dd"} Nov 02 20:45:41.746973 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:41.746814 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" event={"ID":"9525e1da-30ce-46db-8588-8f67c46da099","Type":"ContainerStarted","Data":"9cb6331398dac41ce382f32d59116dff2bedc2d1e5b686fb67e598eab579713b"} Nov 02 20:45:41.746973 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:41.746875 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" event={"ID":"9525e1da-30ce-46db-8588-8f67c46da099","Type":"ContainerStarted","Data":"01368f0211d8d4e272f21c545d4e6ef28cc5ad491fc7cdc403ab33ed99325ac8"} Nov 02 20:45:41.747124 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:41.747093 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" Nov 02 20:45:41.864546 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:41.864520 4971 scope.go:122] "RemoveContainer" containerID="7f19b2f501711fa531d7a440847f47a00e4c1b9d259a480c1318c496b475472b" Nov 02 20:45:41.874986 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:41.874965 4971 scope.go:122] "RemoveContainer" containerID="56aefc2a2b9d0f33cc4404cf15961ca3160e2a9f2f813e846612c38194a84dc3" Nov 02 20:45:41.884891 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:41.884874 4971 scope.go:122] "RemoveContainer" containerID="13d64a689beb6f8e7442e7192443217c4dae70ec64a3f2499d2482808181ede1" Nov 02 20:45:42.066808 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.066762 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" podStartSLOduration=3.066749601 podStartE2EDuration="3.066749601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:42.062871668 +0000 UTC m=+1870.352528197" watchObservedRunningTime="2026-11-02 20:45:42.066749601 +0000 UTC m=+1870.356406129" Nov 02 20:45:42.174165 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.174108 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" podStartSLOduration=3.174071668 podStartE2EDuration="3.174071668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:42.171438725 +0000 UTC m=+1870.461095253" watchObservedRunningTime="2026-11-02 20:45:42.174071668 +0000 UTC m=+1870.463728197" Nov 02 20:45:42.827672 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.827308 4971 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:45:42.827672 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.827380 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" podUID="7a943a64-646e-4533-a37c-71334481ec3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:45:42.944832 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.944808 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg" Nov 02 20:45:42.946710 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.946689 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx" Nov 02 20:45:42.990920 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.990891 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5eee46-301a-46eb-8672-458a4b62a3ec-kube-api-access-z8x5l\" (UniqueName: \"kubernetes.io/projected/3b5eee46-301a-46eb-8672-458a4b62a3ec-kube-api-access-z8x5l\") pod \"3b5eee46-301a-46eb-8672-458a4b62a3ec\" (UID: \"3b5eee46-301a-46eb-8672-458a4b62a3ec\") " Nov 02 20:45:42.991040 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.990980 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-maas-api-xks-ca\") pod \"42bde7ba-8b52-46dc-98a4-ae6df715ceb0\" (UID: \"42bde7ba-8b52-46dc-98a4-ae6df715ceb0\") " Nov 02 20:45:42.991040 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.991016 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5eee46-301a-46eb-8672-458a4b62a3ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5eee46-301a-46eb-8672-458a4b62a3ec-maas-api-openshift-ca\") pod \"3b5eee46-301a-46eb-8672-458a4b62a3ec\" (UID: \"3b5eee46-301a-46eb-8672-458a4b62a3ec\") " Nov 02 20:45:42.991198 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.991123 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-kube-api-access-76qr4\" (UniqueName: \"kubernetes.io/projected/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-kube-api-access-76qr4\") pod \"42bde7ba-8b52-46dc-98a4-ae6df715ceb0\" (UID: \"42bde7ba-8b52-46dc-98a4-ae6df715ceb0\") " Nov 02 20:45:42.991264 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.991199 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-maas-api-openshift-ca\") pod \"42bde7ba-8b52-46dc-98a4-ae6df715ceb0\" (UID: \"42bde7ba-8b52-46dc-98a4-ae6df715ceb0\") " Nov 02 20:45:42.991320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.991286 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b5eee46-301a-46eb-8672-458a4b62a3ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5eee46-301a-46eb-8672-458a4b62a3ec-maas-api-xks-ca\") pod \"3b5eee46-301a-46eb-8672-458a4b62a3ec\" (UID: \"3b5eee46-301a-46eb-8672-458a4b62a3ec\") " Nov 02 20:45:42.991437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.991408 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5eee46-301a-46eb-8672-458a4b62a3ec-maas-api-openshift-ca" pod "3b5eee46-301a-46eb-8672-458a4b62a3ec" (UID: "3b5eee46-301a-46eb-8672-458a4b62a3ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:42.991630 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.991612 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-maas-api-openshift-ca" pod "42bde7ba-8b52-46dc-98a4-ae6df715ceb0" (UID: "42bde7ba-8b52-46dc-98a4-ae6df715ceb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:42.991937 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.991914 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5eee46-301a-46eb-8672-458a4b62a3ec-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:42.992046 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.991941 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:42.993463 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.993438 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-maas-api-xks-ca" pod "42bde7ba-8b52-46dc-98a4-ae6df715ceb0" (UID: "42bde7ba-8b52-46dc-98a4-ae6df715ceb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:45:42.993566 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.993534 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-kube-api-access-76qr4" pod "42bde7ba-8b52-46dc-98a4-ae6df715ceb0" (UID: "42bde7ba-8b52-46dc-98a4-ae6df715ceb0"). InnerVolumeSpecName "kube-api-access-76qr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:42.993657 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.993635 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5eee46-301a-46eb-8672-458a4b62a3ec-kube-api-access-z8x5l" pod "3b5eee46-301a-46eb-8672-458a4b62a3ec" (UID: "3b5eee46-301a-46eb-8672-458a4b62a3ec"). InnerVolumeSpecName "kube-api-access-z8x5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:42.993715 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:42.993662 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b5eee46-301a-46eb-8672-458a4b62a3ec-maas-api-xks-ca" pod "3b5eee46-301a-46eb-8672-458a4b62a3ec" (UID: "3b5eee46-301a-46eb-8672-458a4b62a3ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:45:43.093375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:43.093299 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8x5l\" (UniqueName: \"kubernetes.io/projected/3b5eee46-301a-46eb-8672-458a4b62a3ec-kube-api-access-z8x5l\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:43.093375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:43.093336 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:43.093375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:43.093351 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76qr4\" (UniqueName: \"kubernetes.io/projected/42bde7ba-8b52-46dc-98a4-ae6df715ceb0-kube-api-access-76qr4\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:43.093375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:43.093366 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5eee46-301a-46eb-8672-458a4b62a3ec-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:43.763108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:43.763065 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg" Nov 02 20:45:43.763302 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:43.763118 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg" event={"ID":"42bde7ba-8b52-46dc-98a4-ae6df715ceb0","Type":"ContainerDied","Data":"ae6faca71170c8840f9c8a064049398caba6a8e45156011edb306160f53a8d10"} Nov 02 20:45:43.763302 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:43.763180 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae6faca71170c8840f9c8a064049398caba6a8e45156011edb306160f53a8d10" Nov 02 20:45:43.764768 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:43.764748 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx" Nov 02 20:45:43.764853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:43.764751 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx" event={"ID":"3b5eee46-301a-46eb-8672-458a4b62a3ec","Type":"ContainerDied","Data":"7a054e6cadb85b63d236d2c7081cb1822f224912efca241da0f50c280760468b"} Nov 02 20:45:43.764897 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:43.764869 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a054e6cadb85b63d236d2c7081cb1822f224912efca241da0f50c280760468b" Nov 02 20:45:44.787505 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.787423 4971 generic.go:356] "Generic (PLEG): container finished" podID="7a943a64-646e-4533-a37c-71334481ec3a" containerID="199e900c640bece93cdb58672d49d1d38d0b04f0b5ae6c965d3f41e972f210f9" exitCode=0 Nov 02 20:45:44.787913 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.787581 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" event={"ID":"7a943a64-646e-4533-a37c-71334481ec3a","Type":"ContainerDied","Data":"199e900c640bece93cdb58672d49d1d38d0b04f0b5ae6c965d3f41e972f210f9"} Nov 02 20:45:44.878447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.878426 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:45:44.957109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.956611 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a943a64-646e-4533-a37c-71334481ec3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a943a64-646e-4533-a37c-71334481ec3a-istiod-ca-cert\") pod \"7a943a64-646e-4533-a37c-71334481ec3a\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " Nov 02 20:45:44.957350 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.957316 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a943a64-646e-4533-a37c-71334481ec3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a943a64-646e-4533-a37c-71334481ec3a-istio-podinfo\") pod \"7a943a64-646e-4533-a37c-71334481ec3a\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " Nov 02 20:45:44.957512 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.957048 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a943a64-646e-4533-a37c-71334481ec3a-istiod-ca-cert" pod "7a943a64-646e-4533-a37c-71334481ec3a" (UID: "7a943a64-646e-4533-a37c-71334481ec3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:44.957512 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.957368 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-workload-socket\") pod \"7a943a64-646e-4533-a37c-71334481ec3a\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " Nov 02 20:45:44.957512 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.957405 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-credential-socket\") pod \"7a943a64-646e-4533-a37c-71334481ec3a\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " Nov 02 20:45:44.957697 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.957509 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-workload-certs\") pod \"7a943a64-646e-4533-a37c-71334481ec3a\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " Nov 02 20:45:44.957697 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.957677 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-workload-socket" pod "7a943a64-646e-4533-a37c-71334481ec3a" (UID: "7a943a64-646e-4533-a37c-71334481ec3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:44.957697 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.957680 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a943a64-646e-4533-a37c-71334481ec3a-istio-token\" (UniqueName: \"kubernetes.io/projected/7a943a64-646e-4533-a37c-71334481ec3a-istio-token\") pod \"7a943a64-646e-4533-a37c-71334481ec3a\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " Nov 02 20:45:44.957846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.957833 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-workload-certs" pod "7a943a64-646e-4533-a37c-71334481ec3a" (UID: "7a943a64-646e-4533-a37c-71334481ec3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:44.957885 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.957834 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a943a64-646e-4533-a37c-71334481ec3a-kube-api-access-87bj7\" (UniqueName: \"kubernetes.io/projected/7a943a64-646e-4533-a37c-71334481ec3a-kube-api-access-87bj7\") pod \"7a943a64-646e-4533-a37c-71334481ec3a\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " Nov 02 20:45:44.957918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.957903 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-istio-data\") pod \"7a943a64-646e-4533-a37c-71334481ec3a\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " Nov 02 20:45:44.957990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.957965 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-istio-envoy\") pod \"7a943a64-646e-4533-a37c-71334481ec3a\" (UID: \"7a943a64-646e-4533-a37c-71334481ec3a\") " Nov 02 20:45:44.959918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.958662 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a943a64-646e-4533-a37c-71334481ec3a-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:44.959918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.958680 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:44.959918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.958689 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:44.959918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.959896 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-istio-data" pod "7a943a64-646e-4533-a37c-71334481ec3a" (UID: "7a943a64-646e-4533-a37c-71334481ec3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:44.960356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.960328 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-credential-socket" pod "7a943a64-646e-4533-a37c-71334481ec3a" (UID: "7a943a64-646e-4533-a37c-71334481ec3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:44.962447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.962424 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a943a64-646e-4533-a37c-71334481ec3a-istio-token" pod "7a943a64-646e-4533-a37c-71334481ec3a" (UID: "7a943a64-646e-4533-a37c-71334481ec3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:44.962534 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.962462 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a943a64-646e-4533-a37c-71334481ec3a-kube-api-access-87bj7" pod "7a943a64-646e-4533-a37c-71334481ec3a" (UID: "7a943a64-646e-4533-a37c-71334481ec3a"). InnerVolumeSpecName "kube-api-access-87bj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:44.962801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.962778 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a943a64-646e-4533-a37c-71334481ec3a-istio-podinfo" pod "7a943a64-646e-4533-a37c-71334481ec3a" (UID: "7a943a64-646e-4533-a37c-71334481ec3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:45:44.964728 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:44.964697 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-istio-envoy" pod "7a943a64-646e-4533-a37c-71334481ec3a" (UID: "7a943a64-646e-4533-a37c-71334481ec3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:45.059927 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:45.059889 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a943a64-646e-4533-a37c-71334481ec3a-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:45.059927 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:45.059923 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87bj7\" (UniqueName: \"kubernetes.io/projected/7a943a64-646e-4533-a37c-71334481ec3a-kube-api-access-87bj7\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:45.059927 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:45.059935 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:45.060305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:45.059944 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:45.060305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:45.059953 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a943a64-646e-4533-a37c-71334481ec3a-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:45.060305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:45.059961 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a943a64-646e-4533-a37c-71334481ec3a-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:45.395188 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:45.395146 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:45.796474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:45.796391 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" Nov 02 20:45:45.796474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:45.796412 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v" event={"ID":"7a943a64-646e-4533-a37c-71334481ec3a","Type":"ContainerDied","Data":"21c6277bb43c66fce9a1755c964b39f954f7bd37307badc67de329a8a9341069"} Nov 02 20:45:45.797365 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:45.796496 4971 scope.go:122] "RemoveContainer" containerID="199e900c640bece93cdb58672d49d1d38d0b04f0b5ae6c965d3f41e972f210f9" Nov 02 20:45:46.004119 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:46.004064 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v"] Nov 02 20:45:46.008071 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:46.008044 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-ck29v"] Nov 02 20:45:46.112943 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:46.112866 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:47.867583 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:47.867541 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a943a64-646e-4533-a37c-71334481ec3a" path="/var/lib/kubelet/pods/7a943a64-646e-4533-a37c-71334481ec3a/volumes" Nov 02 20:45:51.000351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:51.000317 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:51.304685 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:51.304578 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:52.757515 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:52.757487 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" Nov 02 20:45:52.757920 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:52.757575 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" Nov 02 20:45:53.892533 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:53.892499 4971 generic.go:356] "Generic (PLEG): container finished" podID="b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" containerID="eb1298232bf22e960593cd92a2b66778bf1f24c8f598b3d7eb019b1b644261dd" exitCode=0 Nov 02 20:45:53.892966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:53.892571 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" event={"ID":"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5","Type":"ContainerDied","Data":"eb1298232bf22e960593cd92a2b66778bf1f24c8f598b3d7eb019b1b644261dd"} Nov 02 20:45:53.892966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:53.892962 4971 scope.go:122] "RemoveContainer" containerID="eb1298232bf22e960593cd92a2b66778bf1f24c8f598b3d7eb019b1b644261dd" Nov 02 20:45:53.894483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:53.894448 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:45:53.894615 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:53.894531 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:45:53.894615 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:53.894578 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:45:54.901806 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:54.901771 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" event={"ID":"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5","Type":"ContainerStarted","Data":"c272dc93126737a4b0159ee87442b2e92c7bb4b177cc7add914bfbb4134579e3"} Nov 02 20:45:55.785824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:55.785785 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:45:55.786013 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:55.785936 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 20:45:55.786013 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:55.785985 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" podUID="b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 20:45:55.903347 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:55.903306 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:56.786064 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:56.786020 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 20:45:56.786298 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:56.786121 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" podUID="b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 20:45:57.159422 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:57.159390 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:57.408407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:57.408372 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:57.675762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:57.675723 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:45:57.785619 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:57.785587 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 20:45:57.785785 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:57.785635 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" podUID="b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 20:45:58.785937 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:58.785901 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 20:45:58.786367 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:58.785957 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" podUID="b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 20:45:59.785624 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:59.785589 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 02 20:45:59.785795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:45:59.785644 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" podUID="b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 02 20:46:00.789633 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:00.789604 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:46:00.790201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:00.790071 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:46:00.790855 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:00.790837 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:46:01.019557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:01.019517 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:01.519570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:01.519536 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:03.068556 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.068520 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm"] Nov 02 20:46:03.069211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.069181 4971 cpu_manager.go:414] "containerMap: removing container" podUID="7a943a64-646e-4533-a37c-71334481ec3a" containerName="istio-proxy" Nov 02 20:46:03.069211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.069211 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a943a64-646e-4533-a37c-71334481ec3a" containerName="istio-proxy" Nov 02 20:46:03.069375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.069229 4971 cpu_manager.go:414] "containerMap: removing container" podUID="3b5eee46-301a-46eb-8672-458a4b62a3ec" containerName="revoke-keys" Nov 02 20:46:03.069375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.069234 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5eee46-301a-46eb-8672-458a4b62a3ec" containerName="revoke-keys" Nov 02 20:46:03.069375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.069243 4971 cpu_manager.go:414] "containerMap: removing container" podUID="42bde7ba-8b52-46dc-98a4-ae6df715ceb0" containerName="revoke-keys" Nov 02 20:46:03.069375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.069248 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42bde7ba-8b52-46dc-98a4-ae6df715ceb0" containerName="revoke-keys" Nov 02 20:46:03.069521 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.069429 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a943a64-646e-4533-a37c-71334481ec3a" containerName="istio-proxy" Nov 02 20:46:03.069521 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.069448 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5eee46-301a-46eb-8672-458a4b62a3ec" containerName="revoke-keys" Nov 02 20:46:03.069521 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.069459 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="42bde7ba-8b52-46dc-98a4-ae6df715ceb0" containerName="revoke-keys" Nov 02 20:46:03.074527 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.074506 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm" Nov 02 20:46:03.127061 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.127038 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm"] Nov 02 20:46:03.131841 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.131810 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7afff03-3359-48c9-8ed5-b04a1d513b0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm\" (UID: \"c7afff03-3359-48c9-8ed5-b04a1d513b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm" Nov 02 20:46:03.131994 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.131968 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkxqs\" (UniqueName: \"kubernetes.io/projected/c7afff03-3359-48c9-8ed5-b04a1d513b0b-kube-api-access-bkxqs\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm\" (UID: \"c7afff03-3359-48c9-8ed5-b04a1d513b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm" Nov 02 20:46:03.132203 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.132171 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7afff03-3359-48c9-8ed5-b04a1d513b0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm\" (UID: \"c7afff03-3359-48c9-8ed5-b04a1d513b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm" Nov 02 20:46:03.168492 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.168468 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:46:03.168594 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.168579 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:46:03.233524 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.233496 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7afff03-3359-48c9-8ed5-b04a1d513b0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm\" (UID: \"c7afff03-3359-48c9-8ed5-b04a1d513b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm" Nov 02 20:46:03.233646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.233556 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkxqs\" (UniqueName: \"kubernetes.io/projected/c7afff03-3359-48c9-8ed5-b04a1d513b0b-kube-api-access-bkxqs\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm\" (UID: \"c7afff03-3359-48c9-8ed5-b04a1d513b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm" Nov 02 20:46:03.233646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.233626 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7afff03-3359-48c9-8ed5-b04a1d513b0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm\" (UID: \"c7afff03-3359-48c9-8ed5-b04a1d513b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm" Nov 02 20:46:03.234293 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.234274 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7afff03-3359-48c9-8ed5-b04a1d513b0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm\" (UID: \"c7afff03-3359-48c9-8ed5-b04a1d513b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm" Nov 02 20:46:03.236074 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.236050 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7afff03-3359-48c9-8ed5-b04a1d513b0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm\" (UID: \"c7afff03-3359-48c9-8ed5-b04a1d513b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm" Nov 02 20:46:03.291473 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.291449 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkxqs\" (UniqueName: \"kubernetes.io/projected/c7afff03-3359-48c9-8ed5-b04a1d513b0b-kube-api-access-bkxqs\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm\" (UID: \"c7afff03-3359-48c9-8ed5-b04a1d513b0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm" Nov 02 20:46:03.339486 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:46:03.339414 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:46:03.388578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.388551 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm" Nov 02 20:46:03.766200 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:03.766165 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm"] Nov 02 20:46:03.766464 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:46:03.766442 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7afff03_3359_48c9_8ed5_b04a1d513b0b.slice/crio-ba136472c9d6113fcb313c2449b1cc5a52e9239a8e121267118dd9ea577e976d WatchSource:0}: Error finding container ba136472c9d6113fcb313c2449b1cc5a52e9239a8e121267118dd9ea577e976d: Status 404 returned error can't find the container with id ba136472c9d6113fcb313c2449b1cc5a52e9239a8e121267118dd9ea577e976d Nov 02 20:46:04.019749 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:04.019676 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm" event={"ID":"c7afff03-3359-48c9-8ed5-b04a1d513b0b","Type":"ContainerStarted","Data":"ba136472c9d6113fcb313c2449b1cc5a52e9239a8e121267118dd9ea577e976d"} Nov 02 20:46:05.029253 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:05.029215 4971 generic.go:356] "Generic (PLEG): container finished" podID="c7afff03-3359-48c9-8ed5-b04a1d513b0b" containerID="f2b6488b8df54fdec69d66ad0a67c09e48fd1898d74f98c421f5991f6738ed85" exitCode=0 Nov 02 20:46:05.029663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:05.029328 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm" event={"ID":"c7afff03-3359-48c9-8ed5-b04a1d513b0b","Type":"ContainerDied","Data":"f2b6488b8df54fdec69d66ad0a67c09e48fd1898d74f98c421f5991f6738ed85"} Nov 02 20:46:06.212178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:06.212152 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm" Nov 02 20:46:06.268581 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:06.268553 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7afff03-3359-48c9-8ed5-b04a1d513b0b-kube-api-access-bkxqs\" (UniqueName: \"kubernetes.io/projected/c7afff03-3359-48c9-8ed5-b04a1d513b0b-kube-api-access-bkxqs\") pod \"c7afff03-3359-48c9-8ed5-b04a1d513b0b\" (UID: \"c7afff03-3359-48c9-8ed5-b04a1d513b0b\") " Nov 02 20:46:06.268718 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:06.268706 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7afff03-3359-48c9-8ed5-b04a1d513b0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7afff03-3359-48c9-8ed5-b04a1d513b0b-maas-api-openshift-ca\") pod \"c7afff03-3359-48c9-8ed5-b04a1d513b0b\" (UID: \"c7afff03-3359-48c9-8ed5-b04a1d513b0b\") " Nov 02 20:46:06.268765 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:06.268738 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7afff03-3359-48c9-8ed5-b04a1d513b0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7afff03-3359-48c9-8ed5-b04a1d513b0b-maas-api-xks-ca\") pod \"c7afff03-3359-48c9-8ed5-b04a1d513b0b\" (UID: \"c7afff03-3359-48c9-8ed5-b04a1d513b0b\") " Nov 02 20:46:06.269067 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:06.269040 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7afff03-3359-48c9-8ed5-b04a1d513b0b-maas-api-openshift-ca" pod "c7afff03-3359-48c9-8ed5-b04a1d513b0b" (UID: "c7afff03-3359-48c9-8ed5-b04a1d513b0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:06.269298 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:06.269278 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7afff03-3359-48c9-8ed5-b04a1d513b0b-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:06.270796 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:06.270770 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7afff03-3359-48c9-8ed5-b04a1d513b0b-kube-api-access-bkxqs" pod "c7afff03-3359-48c9-8ed5-b04a1d513b0b" (UID: "c7afff03-3359-48c9-8ed5-b04a1d513b0b"). InnerVolumeSpecName "kube-api-access-bkxqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:06.270946 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:06.270922 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7afff03-3359-48c9-8ed5-b04a1d513b0b-maas-api-xks-ca" pod "c7afff03-3359-48c9-8ed5-b04a1d513b0b" (UID: "c7afff03-3359-48c9-8ed5-b04a1d513b0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:46:06.358255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:06.358215 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:06.370749 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:06.370721 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7afff03-3359-48c9-8ed5-b04a1d513b0b-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:06.370749 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:06.370749 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkxqs\" (UniqueName: \"kubernetes.io/projected/c7afff03-3359-48c9-8ed5-b04a1d513b0b-kube-api-access-bkxqs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:07.044772 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:07.044746 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm" Nov 02 20:46:07.044941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:07.044778 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm" event={"ID":"c7afff03-3359-48c9-8ed5-b04a1d513b0b","Type":"ContainerDied","Data":"ba136472c9d6113fcb313c2449b1cc5a52e9239a8e121267118dd9ea577e976d"} Nov 02 20:46:07.044941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:07.044824 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba136472c9d6113fcb313c2449b1cc5a52e9239a8e121267118dd9ea577e976d" Nov 02 20:46:11.467786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:11.467736 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:15.015525 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:15.015479 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:15.263060 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:15.263024 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:15.711204 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:15.711167 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:15.996850 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:15.996739 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:16.262612 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:16.262512 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:17.279054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:17.279015 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:17.536733 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:17.536642 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:17.680317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:17.680286 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn"] Nov 02 20:46:17.726347 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:17.726319 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bce768-qkxzn"] Nov 02 20:46:17.789415 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:17.789353 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:17.868864 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:17.868834 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64210fb9-2c46-4cbc-b8fe-b17cf04ce960" path="/var/lib/kubelet/pods/64210fb9-2c46-4cbc-b8fe-b17cf04ce960/volumes" Nov 02 20:46:18.389794 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:18.389757 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:22.444553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:22.444510 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:22.690458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:22.690427 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:24.513374 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:24.513331 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:24.746590 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:24.746558 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6"] Nov 02 20:46:24.746822 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:24.746801 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" podUID="ebf56dbf-ece9-4da3-9589-77c972d8552e" containerName="istio-proxy" containerID="cri-o://3e386a4b3a9b03dc05d87e737a9b9b44130d8cf45f07328608b3a081d5b2763b" gracePeriod=30 Nov 02 20:46:24.784382 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:24.784325 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z"] Nov 02 20:46:24.784536 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:24.784503 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" podUID="01ef03b8-5cfb-472d-9dff-31794d02143b" containerName="istio-proxy" containerID="cri-o://e3395ae279d10874aee4dfdfb3d55cff2acc64ff9917a484604d1d28a47d3d29" gracePeriod=30 Nov 02 20:46:24.789402 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:24.789376 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:25.057462 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:25.057370 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:26.048563 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:26.048530 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q"] Nov 02 20:46:26.094709 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:26.094676 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76782b5d-wxk4q"] Nov 02 20:46:27.867965 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:27.867930 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5e7e19d-06dd-46c1-b35d-645c9f1f02f3" path="/var/lib/kubelet/pods/b5e7e19d-06dd-46c1-b35d-645c9f1f02f3/volumes" Nov 02 20:46:27.970868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:27.970833 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:28.068904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:28.068870 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:46:28.069132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:28.068928 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" podUID="ebf56dbf-ece9-4da3-9589-77c972d8552e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:46:29.508566 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:29.508528 4971 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:46:29.508946 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:29.508596 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" podUID="01ef03b8-5cfb-472d-9dff-31794d02143b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:46:30.107783 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.107760 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:46:30.136587 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.136566 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:46:30.222137 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.222070 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-data\") pod \"01ef03b8-5cfb-472d-9dff-31794d02143b\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " Nov 02 20:46:30.222302 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.222171 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-credential-socket\") pod \"01ef03b8-5cfb-472d-9dff-31794d02143b\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " Nov 02 20:46:30.222302 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.222212 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-credential-socket\") pod \"ebf56dbf-ece9-4da3-9589-77c972d8552e\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " Nov 02 20:46:30.222447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.222324 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-token\" (UniqueName: \"kubernetes.io/projected/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-token\") pod \"ebf56dbf-ece9-4da3-9589-77c972d8552e\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " Nov 02 20:46:30.222447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.222368 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-workload-socket\") pod \"01ef03b8-5cfb-472d-9dff-31794d02143b\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " Nov 02 20:46:30.222447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.222354 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-data" pod "01ef03b8-5cfb-472d-9dff-31794d02143b" (UID: "01ef03b8-5cfb-472d-9dff-31794d02143b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:30.222447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.222419 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-podinfo\") pod \"01ef03b8-5cfb-472d-9dff-31794d02143b\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " Nov 02 20:46:30.222447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.222432 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-credential-socket" pod "01ef03b8-5cfb-472d-9dff-31794d02143b" (UID: "01ef03b8-5cfb-472d-9dff-31794d02143b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:30.222733 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.222447 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-envoy\") pod \"01ef03b8-5cfb-472d-9dff-31794d02143b\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " Nov 02 20:46:30.222733 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.222470 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-credential-socket" pod "ebf56dbf-ece9-4da3-9589-77c972d8552e" (UID: "ebf56dbf-ece9-4da3-9589-77c972d8552e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:30.222733 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.222490 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-workload-socket\") pod \"ebf56dbf-ece9-4da3-9589-77c972d8552e\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " Nov 02 20:46:30.222733 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.222719 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-workload-socket" pod "01ef03b8-5cfb-472d-9dff-31794d02143b" (UID: "01ef03b8-5cfb-472d-9dff-31794d02143b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:30.222954 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.222850 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-workload-certs\") pod \"01ef03b8-5cfb-472d-9dff-31794d02143b\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " Nov 02 20:46:30.222954 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.222892 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ef03b8-5cfb-472d-9dff-31794d02143b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ef03b8-5cfb-472d-9dff-31794d02143b-istiod-ca-cert\") pod \"01ef03b8-5cfb-472d-9dff-31794d02143b\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " Nov 02 20:46:30.222954 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.222914 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-workload-socket" pod "ebf56dbf-ece9-4da3-9589-77c972d8552e" (UID: "ebf56dbf-ece9-4da3-9589-77c972d8552e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:30.222954 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.222922 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-envoy\") pod \"ebf56dbf-ece9-4da3-9589-77c972d8552e\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " Nov 02 20:46:30.223312 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.223003 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ef03b8-5cfb-472d-9dff-31794d02143b-kube-api-access-bhb2b\" (UniqueName: \"kubernetes.io/projected/01ef03b8-5cfb-472d-9dff-31794d02143b-kube-api-access-bhb2b\") pod \"01ef03b8-5cfb-472d-9dff-31794d02143b\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " Nov 02 20:46:30.223312 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.223037 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-token\" (UniqueName: \"kubernetes.io/projected/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-token\") pod \"01ef03b8-5cfb-472d-9dff-31794d02143b\" (UID: \"01ef03b8-5cfb-472d-9dff-31794d02143b\") " Nov 02 20:46:30.223312 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.223120 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-podinfo\") pod \"ebf56dbf-ece9-4da3-9589-77c972d8552e\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " Nov 02 20:46:30.223312 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.223170 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-data\") pod \"ebf56dbf-ece9-4da3-9589-77c972d8552e\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " Nov 02 20:46:30.223538 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.223388 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-workload-certs\") pod \"ebf56dbf-ece9-4da3-9589-77c972d8552e\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " Nov 02 20:46:30.223538 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.223464 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebf56dbf-ece9-4da3-9589-77c972d8552e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebf56dbf-ece9-4da3-9589-77c972d8552e-istiod-ca-cert\") pod \"ebf56dbf-ece9-4da3-9589-77c972d8552e\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " Nov 02 20:46:30.223538 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.223510 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf56dbf-ece9-4da3-9589-77c972d8552e-kube-api-access-wdwz4\" (UniqueName: \"kubernetes.io/projected/ebf56dbf-ece9-4da3-9589-77c972d8552e-kube-api-access-wdwz4\") pod \"ebf56dbf-ece9-4da3-9589-77c972d8552e\" (UID: \"ebf56dbf-ece9-4da3-9589-77c972d8552e\") " Nov 02 20:46:30.225277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.224157 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.225277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.224183 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.225277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.224199 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.225277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.224215 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.225277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.224229 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.225277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.224744 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-workload-certs" pod "01ef03b8-5cfb-472d-9dff-31794d02143b" (UID: "01ef03b8-5cfb-472d-9dff-31794d02143b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:30.225277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.225055 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ef03b8-5cfb-472d-9dff-31794d02143b-istiod-ca-cert" pod "01ef03b8-5cfb-472d-9dff-31794d02143b" (UID: "01ef03b8-5cfb-472d-9dff-31794d02143b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:30.226200 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.225402 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-data" pod "ebf56dbf-ece9-4da3-9589-77c972d8552e" (UID: "ebf56dbf-ece9-4da3-9589-77c972d8552e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:30.226200 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.225616 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-workload-certs" pod "ebf56dbf-ece9-4da3-9589-77c972d8552e" (UID: "ebf56dbf-ece9-4da3-9589-77c972d8552e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:30.226200 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.225839 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf56dbf-ece9-4da3-9589-77c972d8552e-istiod-ca-cert" pod "ebf56dbf-ece9-4da3-9589-77c972d8552e" (UID: "ebf56dbf-ece9-4da3-9589-77c972d8552e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:30.226815 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.226739 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-token" pod "01ef03b8-5cfb-472d-9dff-31794d02143b" (UID: "01ef03b8-5cfb-472d-9dff-31794d02143b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:30.226815 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.226756 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-podinfo" pod "01ef03b8-5cfb-472d-9dff-31794d02143b" (UID: "01ef03b8-5cfb-472d-9dff-31794d02143b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:46:30.226815 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.226781 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-token" pod "ebf56dbf-ece9-4da3-9589-77c972d8552e" (UID: "ebf56dbf-ece9-4da3-9589-77c972d8552e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:30.227733 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.227707 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ef03b8-5cfb-472d-9dff-31794d02143b-kube-api-access-bhb2b" pod "01ef03b8-5cfb-472d-9dff-31794d02143b" (UID: "01ef03b8-5cfb-472d-9dff-31794d02143b"). InnerVolumeSpecName "kube-api-access-bhb2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:30.227852 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.227765 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-podinfo" pod "ebf56dbf-ece9-4da3-9589-77c972d8552e" (UID: "ebf56dbf-ece9-4da3-9589-77c972d8552e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:46:30.227921 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.227894 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-envoy" pod "ebf56dbf-ece9-4da3-9589-77c972d8552e" (UID: "ebf56dbf-ece9-4da3-9589-77c972d8552e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:30.228980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.228955 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf56dbf-ece9-4da3-9589-77c972d8552e-kube-api-access-wdwz4" pod "ebf56dbf-ece9-4da3-9589-77c972d8552e" (UID: "ebf56dbf-ece9-4da3-9589-77c972d8552e"). InnerVolumeSpecName "kube-api-access-wdwz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:30.229186 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.229165 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-envoy" pod "01ef03b8-5cfb-472d-9dff-31794d02143b" (UID: "01ef03b8-5cfb-472d-9dff-31794d02143b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:30.251902 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.251874 4971 generic.go:356] "Generic (PLEG): container finished" podID="01ef03b8-5cfb-472d-9dff-31794d02143b" containerID="e3395ae279d10874aee4dfdfb3d55cff2acc64ff9917a484604d1d28a47d3d29" exitCode=0 Nov 02 20:46:30.252004 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.251982 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" Nov 02 20:46:30.252070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.252014 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" event={"ID":"01ef03b8-5cfb-472d-9dff-31794d02143b","Type":"ContainerDied","Data":"e3395ae279d10874aee4dfdfb3d55cff2acc64ff9917a484604d1d28a47d3d29"} Nov 02 20:46:30.252145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.252101 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z" event={"ID":"01ef03b8-5cfb-472d-9dff-31794d02143b","Type":"ContainerDied","Data":"1b427bbd588483171937362de909187065d3869432221d47f68f9e34b4d5a436"} Nov 02 20:46:30.252145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.252137 4971 scope.go:122] "RemoveContainer" containerID="e3395ae279d10874aee4dfdfb3d55cff2acc64ff9917a484604d1d28a47d3d29" Nov 02 20:46:30.254065 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.254039 4971 generic.go:356] "Generic (PLEG): container finished" podID="ebf56dbf-ece9-4da3-9589-77c972d8552e" containerID="3e386a4b3a9b03dc05d87e737a9b9b44130d8cf45f07328608b3a081d5b2763b" exitCode=0 Nov 02 20:46:30.254220 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.254108 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" Nov 02 20:46:30.254220 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.254144 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" event={"ID":"ebf56dbf-ece9-4da3-9589-77c972d8552e","Type":"ContainerDied","Data":"3e386a4b3a9b03dc05d87e737a9b9b44130d8cf45f07328608b3a081d5b2763b"} Nov 02 20:46:30.254220 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.254215 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6" event={"ID":"ebf56dbf-ece9-4da3-9589-77c972d8552e","Type":"ContainerDied","Data":"6d2531efca5d0c2af972a8ec401ce1f5df93299d4ed85a48184db03d276652ec"} Nov 02 20:46:30.267857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.267839 4971 scope.go:122] "RemoveContainer" containerID="e3395ae279d10874aee4dfdfb3d55cff2acc64ff9917a484604d1d28a47d3d29" Nov 02 20:46:30.268579 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:46:30.268560 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3395ae279d10874aee4dfdfb3d55cff2acc64ff9917a484604d1d28a47d3d29\": container with ID starting with e3395ae279d10874aee4dfdfb3d55cff2acc64ff9917a484604d1d28a47d3d29 not found: ID does not exist" containerID="e3395ae279d10874aee4dfdfb3d55cff2acc64ff9917a484604d1d28a47d3d29" Nov 02 20:46:30.268681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.268590 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3395ae279d10874aee4dfdfb3d55cff2acc64ff9917a484604d1d28a47d3d29"} err="failed to get container status \"e3395ae279d10874aee4dfdfb3d55cff2acc64ff9917a484604d1d28a47d3d29\": rpc error: code = NotFound desc = could not find container \"e3395ae279d10874aee4dfdfb3d55cff2acc64ff9917a484604d1d28a47d3d29\": container with ID starting with e3395ae279d10874aee4dfdfb3d55cff2acc64ff9917a484604d1d28a47d3d29 not found: ID does not exist" Nov 02 20:46:30.268681 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.268615 4971 scope.go:122] "RemoveContainer" containerID="3e386a4b3a9b03dc05d87e737a9b9b44130d8cf45f07328608b3a081d5b2763b" Nov 02 20:46:30.281144 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.281066 4971 scope.go:122] "RemoveContainer" containerID="3e386a4b3a9b03dc05d87e737a9b9b44130d8cf45f07328608b3a081d5b2763b" Nov 02 20:46:30.281411 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:46:30.281391 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e386a4b3a9b03dc05d87e737a9b9b44130d8cf45f07328608b3a081d5b2763b\": container with ID starting with 3e386a4b3a9b03dc05d87e737a9b9b44130d8cf45f07328608b3a081d5b2763b not found: ID does not exist" containerID="3e386a4b3a9b03dc05d87e737a9b9b44130d8cf45f07328608b3a081d5b2763b" Nov 02 20:46:30.281488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.281415 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e386a4b3a9b03dc05d87e737a9b9b44130d8cf45f07328608b3a081d5b2763b"} err="failed to get container status \"3e386a4b3a9b03dc05d87e737a9b9b44130d8cf45f07328608b3a081d5b2763b\": rpc error: code = NotFound desc = could not find container \"3e386a4b3a9b03dc05d87e737a9b9b44130d8cf45f07328608b3a081d5b2763b\": container with ID starting with 3e386a4b3a9b03dc05d87e737a9b9b44130d8cf45f07328608b3a081d5b2763b not found: ID does not exist" Nov 02 20:46:30.326433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.326381 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebf56dbf-ece9-4da3-9589-77c972d8552e-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.326433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.326404 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdwz4\" (UniqueName: \"kubernetes.io/projected/ebf56dbf-ece9-4da3-9589-77c972d8552e-kube-api-access-wdwz4\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.326433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.326413 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.326433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.326422 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.326433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.326431 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.326433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.326440 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ef03b8-5cfb-472d-9dff-31794d02143b-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.326708 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.326453 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ef03b8-5cfb-472d-9dff-31794d02143b-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.326708 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.326466 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.326708 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.326477 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhb2b\" (UniqueName: \"kubernetes.io/projected/01ef03b8-5cfb-472d-9dff-31794d02143b-kube-api-access-bhb2b\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.326708 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.326486 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ef03b8-5cfb-472d-9dff-31794d02143b-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.326708 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.326495 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.326708 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.326504 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.326708 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.326512 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebf56dbf-ece9-4da3-9589-77c972d8552e-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:30.377437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.377413 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6"] Nov 02 20:46:30.381255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.381234 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jr2c6"] Nov 02 20:46:30.547369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.547339 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z"] Nov 02 20:46:30.551570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:30.551545 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-9s76z"] Nov 02 20:46:31.875265 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:31.875228 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ef03b8-5cfb-472d-9dff-31794d02143b" path="/var/lib/kubelet/pods/01ef03b8-5cfb-472d-9dff-31794d02143b/volumes" Nov 02 20:46:31.875686 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:31.875671 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebf56dbf-ece9-4da3-9589-77c972d8552e" path="/var/lib/kubelet/pods/ebf56dbf-ece9-4da3-9589-77c972d8552e/volumes" Nov 02 20:46:32.740912 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:32.740881 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:33.151724 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:33.151690 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:33.340064 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:46:33.340030 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:46:38.195756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:38.195717 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:38.446739 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:38.446666 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:39.910640 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:39.910603 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg"] Nov 02 20:46:39.911142 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:39.910790 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" podUID="697fdff1-eee0-4118-9af2-4b24a48359ea" containerName="istio-proxy" containerID="cri-o://60b388975fbc63c6da8f2c956800e1249ba245b9027643adc3ca7d6f0ade175e" gracePeriod=30 Nov 02 20:46:42.010941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:42.010858 4971 scope.go:122] "RemoveContainer" containerID="96451e895c08c479094171904e566fc410f3fce78494a51567893026356c199f" Nov 02 20:46:42.291049 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:42.290975 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:46:42.291222 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:42.291050 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" podUID="697fdff1-eee0-4118-9af2-4b24a48359ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:46:42.996031 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:42.995984 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:43.423551 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:43.423518 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:44.139260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:44.139222 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:44.522253 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:44.522160 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:44.835471 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:44.835381 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:45.036568 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.036538 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm"] Nov 02 20:46:45.082782 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.082748 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-654ae4-llwfm"] Nov 02 20:46:45.187136 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.187112 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:46:45.219278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.219251 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-podinfo\") pod \"697fdff1-eee0-4118-9af2-4b24a48359ea\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " Nov 02 20:46:45.219420 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.219296 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-envoy\") pod \"697fdff1-eee0-4118-9af2-4b24a48359ea\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " Nov 02 20:46:45.219420 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.219413 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-token\" (UniqueName: \"kubernetes.io/projected/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-token\") pod \"697fdff1-eee0-4118-9af2-4b24a48359ea\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " Nov 02 20:46:45.219556 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.219497 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-workload-socket\") pod \"697fdff1-eee0-4118-9af2-4b24a48359ea\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " Nov 02 20:46:45.219620 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.219564 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-credential-socket\") pod \"697fdff1-eee0-4118-9af2-4b24a48359ea\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " Nov 02 20:46:45.219675 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.219627 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-data\") pod \"697fdff1-eee0-4118-9af2-4b24a48359ea\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " Nov 02 20:46:45.219726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.219679 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-workload-certs\") pod \"697fdff1-eee0-4118-9af2-4b24a48359ea\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " Nov 02 20:46:45.219790 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.219720 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/697fdff1-eee0-4118-9af2-4b24a48359ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/697fdff1-eee0-4118-9af2-4b24a48359ea-istiod-ca-cert\") pod \"697fdff1-eee0-4118-9af2-4b24a48359ea\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " Nov 02 20:46:45.219790 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.219752 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-workload-socket" pod "697fdff1-eee0-4118-9af2-4b24a48359ea" (UID: "697fdff1-eee0-4118-9af2-4b24a48359ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:45.219955 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.219934 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-data" pod "697fdff1-eee0-4118-9af2-4b24a48359ea" (UID: "697fdff1-eee0-4118-9af2-4b24a48359ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:45.220260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.220234 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-workload-certs" pod "697fdff1-eee0-4118-9af2-4b24a48359ea" (UID: "697fdff1-eee0-4118-9af2-4b24a48359ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:45.220356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.220333 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/697fdff1-eee0-4118-9af2-4b24a48359ea-istiod-ca-cert" pod "697fdff1-eee0-4118-9af2-4b24a48359ea" (UID: "697fdff1-eee0-4118-9af2-4b24a48359ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:45.220551 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.220531 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:45.220633 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.220557 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:45.220633 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.220572 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/697fdff1-eee0-4118-9af2-4b24a48359ea-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:45.220633 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.220586 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:45.220735 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.220647 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-credential-socket" pod "697fdff1-eee0-4118-9af2-4b24a48359ea" (UID: "697fdff1-eee0-4118-9af2-4b24a48359ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:45.221657 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.221632 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-envoy" pod "697fdff1-eee0-4118-9af2-4b24a48359ea" (UID: "697fdff1-eee0-4118-9af2-4b24a48359ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:45.221725 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.221676 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-podinfo" pod "697fdff1-eee0-4118-9af2-4b24a48359ea" (UID: "697fdff1-eee0-4118-9af2-4b24a48359ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:46:45.221858 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.221841 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-token" pod "697fdff1-eee0-4118-9af2-4b24a48359ea" (UID: "697fdff1-eee0-4118-9af2-4b24a48359ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:45.321942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.321917 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/697fdff1-eee0-4118-9af2-4b24a48359ea-kube-api-access-8d27n\" (UniqueName: \"kubernetes.io/projected/697fdff1-eee0-4118-9af2-4b24a48359ea-kube-api-access-8d27n\") pod \"697fdff1-eee0-4118-9af2-4b24a48359ea\" (UID: \"697fdff1-eee0-4118-9af2-4b24a48359ea\") " Nov 02 20:46:45.322907 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.322844 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:45.323966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.323583 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:45.323966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.323963 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/697fdff1-eee0-4118-9af2-4b24a48359ea-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:45.324195 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.323984 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/697fdff1-eee0-4118-9af2-4b24a48359ea-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:45.325135 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.324410 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697fdff1-eee0-4118-9af2-4b24a48359ea-kube-api-access-8d27n" pod "697fdff1-eee0-4118-9af2-4b24a48359ea" (UID: "697fdff1-eee0-4118-9af2-4b24a48359ea"). InnerVolumeSpecName "kube-api-access-8d27n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:45.395894 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.395863 4971 generic.go:356] "Generic (PLEG): container finished" podID="697fdff1-eee0-4118-9af2-4b24a48359ea" containerID="60b388975fbc63c6da8f2c956800e1249ba245b9027643adc3ca7d6f0ade175e" exitCode=0 Nov 02 20:46:45.396056 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.395943 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" Nov 02 20:46:45.396056 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.395945 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" event={"ID":"697fdff1-eee0-4118-9af2-4b24a48359ea","Type":"ContainerDied","Data":"60b388975fbc63c6da8f2c956800e1249ba245b9027643adc3ca7d6f0ade175e"} Nov 02 20:46:45.396056 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.396034 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg" event={"ID":"697fdff1-eee0-4118-9af2-4b24a48359ea","Type":"ContainerDied","Data":"a4fb3555ac8f739787c8faaa5304e2664caa9679c637746898d1bcff5cd896be"} Nov 02 20:46:45.396337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.396061 4971 scope.go:122] "RemoveContainer" containerID="60b388975fbc63c6da8f2c956800e1249ba245b9027643adc3ca7d6f0ade175e" Nov 02 20:46:45.407196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.407181 4971 scope.go:122] "RemoveContainer" containerID="60b388975fbc63c6da8f2c956800e1249ba245b9027643adc3ca7d6f0ade175e" Nov 02 20:46:45.407492 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:46:45.407472 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b388975fbc63c6da8f2c956800e1249ba245b9027643adc3ca7d6f0ade175e\": container with ID starting with 60b388975fbc63c6da8f2c956800e1249ba245b9027643adc3ca7d6f0ade175e not found: ID does not exist" containerID="60b388975fbc63c6da8f2c956800e1249ba245b9027643adc3ca7d6f0ade175e" Nov 02 20:46:45.407535 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.407501 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b388975fbc63c6da8f2c956800e1249ba245b9027643adc3ca7d6f0ade175e"} err="failed to get container status \"60b388975fbc63c6da8f2c956800e1249ba245b9027643adc3ca7d6f0ade175e\": rpc error: code = NotFound desc = could not find container \"60b388975fbc63c6da8f2c956800e1249ba245b9027643adc3ca7d6f0ade175e\": container with ID starting with 60b388975fbc63c6da8f2c956800e1249ba245b9027643adc3ca7d6f0ade175e not found: ID does not exist" Nov 02 20:46:45.425178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.425145 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d27n\" (UniqueName: \"kubernetes.io/projected/697fdff1-eee0-4118-9af2-4b24a48359ea-kube-api-access-8d27n\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:45.470606 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.470577 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv"] Nov 02 20:46:45.471329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.471311 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c7afff03-3359-48c9-8ed5-b04a1d513b0b" containerName="revoke-keys" Nov 02 20:46:45.471329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.471329 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7afff03-3359-48c9-8ed5-b04a1d513b0b" containerName="revoke-keys" Nov 02 20:46:45.471490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.471349 4971 cpu_manager.go:414] "containerMap: removing container" podUID="697fdff1-eee0-4118-9af2-4b24a48359ea" containerName="istio-proxy" Nov 02 20:46:45.471490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.471355 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="697fdff1-eee0-4118-9af2-4b24a48359ea" containerName="istio-proxy" Nov 02 20:46:45.471490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.471368 4971 cpu_manager.go:414] "containerMap: removing container" podUID="01ef03b8-5cfb-472d-9dff-31794d02143b" containerName="istio-proxy" Nov 02 20:46:45.471490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.471373 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ef03b8-5cfb-472d-9dff-31794d02143b" containerName="istio-proxy" Nov 02 20:46:45.471490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.471380 4971 cpu_manager.go:414] "containerMap: removing container" podUID="ebf56dbf-ece9-4da3-9589-77c972d8552e" containerName="istio-proxy" Nov 02 20:46:45.471490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.471384 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf56dbf-ece9-4da3-9589-77c972d8552e" containerName="istio-proxy" Nov 02 20:46:45.471490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.471470 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="697fdff1-eee0-4118-9af2-4b24a48359ea" containerName="istio-proxy" Nov 02 20:46:45.471490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.471481 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ef03b8-5cfb-472d-9dff-31794d02143b" containerName="istio-proxy" Nov 02 20:46:45.471490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.471486 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf56dbf-ece9-4da3-9589-77c972d8552e" containerName="istio-proxy" Nov 02 20:46:45.471930 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.471498 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7afff03-3359-48c9-8ed5-b04a1d513b0b" containerName="revoke-keys" Nov 02 20:46:45.476564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.476543 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv" Nov 02 20:46:45.526522 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.526496 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae015af-a1f4-44a1-9fc0-0b551d21ac42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv\" (UID: \"dae015af-a1f4-44a1-9fc0-0b551d21ac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv" Nov 02 20:46:45.526910 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.526546 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae015af-a1f4-44a1-9fc0-0b551d21ac42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv\" (UID: \"dae015af-a1f4-44a1-9fc0-0b551d21ac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv" Nov 02 20:46:45.526910 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.526735 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j52rc\" (UniqueName: \"kubernetes.io/projected/dae015af-a1f4-44a1-9fc0-0b551d21ac42-kube-api-access-j52rc\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv\" (UID: \"dae015af-a1f4-44a1-9fc0-0b551d21ac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv" Nov 02 20:46:45.528148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.528129 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:46:45.528257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.528161 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:46:45.532574 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.532556 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv"] Nov 02 20:46:45.557410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.557389 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg"] Nov 02 20:46:45.558956 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.558929 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:45.562859 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.562840 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-x5pvg"] Nov 02 20:46:45.628070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.628045 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j52rc\" (UniqueName: \"kubernetes.io/projected/dae015af-a1f4-44a1-9fc0-0b551d21ac42-kube-api-access-j52rc\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv\" (UID: \"dae015af-a1f4-44a1-9fc0-0b551d21ac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv" Nov 02 20:46:45.628191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.628163 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae015af-a1f4-44a1-9fc0-0b551d21ac42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv\" (UID: \"dae015af-a1f4-44a1-9fc0-0b551d21ac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv" Nov 02 20:46:45.628235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.628191 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae015af-a1f4-44a1-9fc0-0b551d21ac42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv\" (UID: \"dae015af-a1f4-44a1-9fc0-0b551d21ac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv" Nov 02 20:46:45.628792 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.628764 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae015af-a1f4-44a1-9fc0-0b551d21ac42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv\" (UID: \"dae015af-a1f4-44a1-9fc0-0b551d21ac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv" Nov 02 20:46:45.630674 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.630654 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae015af-a1f4-44a1-9fc0-0b551d21ac42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv\" (UID: \"dae015af-a1f4-44a1-9fc0-0b551d21ac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv" Nov 02 20:46:45.685816 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.685760 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j52rc\" (UniqueName: \"kubernetes.io/projected/dae015af-a1f4-44a1-9fc0-0b551d21ac42-kube-api-access-j52rc\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv\" (UID: \"dae015af-a1f4-44a1-9fc0-0b551d21ac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv" Nov 02 20:46:45.790768 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.790741 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv" Nov 02 20:46:45.873260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.873225 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="697fdff1-eee0-4118-9af2-4b24a48359ea" path="/var/lib/kubelet/pods/697fdff1-eee0-4118-9af2-4b24a48359ea/volumes" Nov 02 20:46:45.873936 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:45.873911 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7afff03-3359-48c9-8ed5-b04a1d513b0b" path="/var/lib/kubelet/pods/c7afff03-3359-48c9-8ed5-b04a1d513b0b/volumes" Nov 02 20:46:46.093820 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:46.093789 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:46.178432 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:46.178404 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv"] Nov 02 20:46:46.179257 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:46:46.179230 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae015af_a1f4_44a1_9fc0_0b551d21ac42.slice/crio-595bd1fb97375845101e0f469ec7d48d62ffd723aee25d3c11c2e6f6ebda3c78 WatchSource:0}: Error finding container 595bd1fb97375845101e0f469ec7d48d62ffd723aee25d3c11c2e6f6ebda3c78: Status 404 returned error can't find the container with id 595bd1fb97375845101e0f469ec7d48d62ffd723aee25d3c11c2e6f6ebda3c78 Nov 02 20:46:46.361191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:46.361155 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:46.404902 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:46.404871 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv" event={"ID":"dae015af-a1f4-44a1-9fc0-0b551d21ac42","Type":"ContainerStarted","Data":"595bd1fb97375845101e0f469ec7d48d62ffd723aee25d3c11c2e6f6ebda3c78"} Nov 02 20:46:46.510790 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:46.510758 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2"] Nov 02 20:46:46.517439 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:46.517413 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c49693-2w7n2"] Nov 02 20:46:46.619553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:46.618912 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:46.994865 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:46.994791 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:47.414598 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:47.414567 4971 generic.go:356] "Generic (PLEG): container finished" podID="dae015af-a1f4-44a1-9fc0-0b551d21ac42" containerID="52b7b2d062f17d2ace656ae751e80f1098db3a4228b6132270da2751bc454ae5" exitCode=0 Nov 02 20:46:47.414797 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:47.414659 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv" event={"ID":"dae015af-a1f4-44a1-9fc0-0b551d21ac42","Type":"ContainerDied","Data":"52b7b2d062f17d2ace656ae751e80f1098db3a4228b6132270da2751bc454ae5"} Nov 02 20:46:47.867293 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:47.867267 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2563c4e-0fe5-453d-b156-d7e11d9025b2" path="/var/lib/kubelet/pods/c2563c4e-0fe5-453d-b156-d7e11d9025b2/volumes" Nov 02 20:46:48.610817 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:48.610794 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv" Nov 02 20:46:48.674132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:48.674071 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dae015af-a1f4-44a1-9fc0-0b551d21ac42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae015af-a1f4-44a1-9fc0-0b551d21ac42-maas-api-xks-ca\") pod \"dae015af-a1f4-44a1-9fc0-0b551d21ac42\" (UID: \"dae015af-a1f4-44a1-9fc0-0b551d21ac42\") " Nov 02 20:46:48.674267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:48.674225 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae015af-a1f4-44a1-9fc0-0b551d21ac42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae015af-a1f4-44a1-9fc0-0b551d21ac42-maas-api-openshift-ca\") pod \"dae015af-a1f4-44a1-9fc0-0b551d21ac42\" (UID: \"dae015af-a1f4-44a1-9fc0-0b551d21ac42\") " Nov 02 20:46:48.674347 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:48.674331 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae015af-a1f4-44a1-9fc0-0b551d21ac42-kube-api-access-j52rc\" (UniqueName: \"kubernetes.io/projected/dae015af-a1f4-44a1-9fc0-0b551d21ac42-kube-api-access-j52rc\") pod \"dae015af-a1f4-44a1-9fc0-0b551d21ac42\" (UID: \"dae015af-a1f4-44a1-9fc0-0b551d21ac42\") " Nov 02 20:46:48.674623 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:48.674600 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae015af-a1f4-44a1-9fc0-0b551d21ac42-maas-api-openshift-ca" pod "dae015af-a1f4-44a1-9fc0-0b551d21ac42" (UID: "dae015af-a1f4-44a1-9fc0-0b551d21ac42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:48.674867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:48.674851 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae015af-a1f4-44a1-9fc0-0b551d21ac42-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:48.676291 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:48.676263 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae015af-a1f4-44a1-9fc0-0b551d21ac42-maas-api-xks-ca" pod "dae015af-a1f4-44a1-9fc0-0b551d21ac42" (UID: "dae015af-a1f4-44a1-9fc0-0b551d21ac42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:46:48.676525 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:48.676503 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae015af-a1f4-44a1-9fc0-0b551d21ac42-kube-api-access-j52rc" pod "dae015af-a1f4-44a1-9fc0-0b551d21ac42" (UID: "dae015af-a1f4-44a1-9fc0-0b551d21ac42"). InnerVolumeSpecName "kube-api-access-j52rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:48.775713 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:48.775682 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j52rc\" (UniqueName: \"kubernetes.io/projected/dae015af-a1f4-44a1-9fc0-0b551d21ac42-kube-api-access-j52rc\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:48.775713 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:48.775716 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae015af-a1f4-44a1-9fc0-0b551d21ac42-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:49.441453 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:49.441423 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv" Nov 02 20:46:49.441899 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:49.441461 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv" event={"ID":"dae015af-a1f4-44a1-9fc0-0b551d21ac42","Type":"ContainerDied","Data":"595bd1fb97375845101e0f469ec7d48d62ffd723aee25d3c11c2e6f6ebda3c78"} Nov 02 20:46:49.441899 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:49.441517 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="595bd1fb97375845101e0f469ec7d48d62ffd723aee25d3c11c2e6f6ebda3c78" Nov 02 20:46:51.257281 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:51.257219 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:51.703177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:51.703119 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:51.950870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:51.950835 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:52.198665 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:52.198633 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:52.800060 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:52.800021 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29"] Nov 02 20:46:52.800994 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:52.800968 4971 cpu_manager.go:414] "containerMap: removing container" podUID="dae015af-a1f4-44a1-9fc0-0b551d21ac42" containerName="revoke-keys" Nov 02 20:46:52.801097 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:52.800996 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae015af-a1f4-44a1-9fc0-0b551d21ac42" containerName="revoke-keys" Nov 02 20:46:52.801158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:52.801147 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae015af-a1f4-44a1-9fc0-0b551d21ac42" containerName="revoke-keys" Nov 02 20:46:52.804574 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:52.804557 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" Nov 02 20:46:52.853529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:52.853507 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:46:52.853658 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:52.853624 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:46:52.904662 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:52.904621 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29"] Nov 02 20:46:52.921826 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:52.921798 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a94273d-8965-422a-b814-0a89bc25fc5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29\" (UID: \"0a94273d-8965-422a-b814-0a89bc25fc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" Nov 02 20:46:52.921963 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:52.921942 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkcnw\" (UniqueName: \"kubernetes.io/projected/0a94273d-8965-422a-b814-0a89bc25fc5c-kube-api-access-lkcnw\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29\" (UID: \"0a94273d-8965-422a-b814-0a89bc25fc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" Nov 02 20:46:52.922047 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:52.922031 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a94273d-8965-422a-b814-0a89bc25fc5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29\" (UID: \"0a94273d-8965-422a-b814-0a89bc25fc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" Nov 02 20:46:53.023121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:53.023055 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a94273d-8965-422a-b814-0a89bc25fc5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29\" (UID: \"0a94273d-8965-422a-b814-0a89bc25fc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" Nov 02 20:46:53.023306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:53.023225 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a94273d-8965-422a-b814-0a89bc25fc5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29\" (UID: \"0a94273d-8965-422a-b814-0a89bc25fc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" Nov 02 20:46:53.023384 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:53.023322 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkcnw\" (UniqueName: \"kubernetes.io/projected/0a94273d-8965-422a-b814-0a89bc25fc5c-kube-api-access-lkcnw\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29\" (UID: \"0a94273d-8965-422a-b814-0a89bc25fc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" Nov 02 20:46:53.023878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:53.023851 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a94273d-8965-422a-b814-0a89bc25fc5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29\" (UID: \"0a94273d-8965-422a-b814-0a89bc25fc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" Nov 02 20:46:53.025799 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:53.025759 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a94273d-8965-422a-b814-0a89bc25fc5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29\" (UID: \"0a94273d-8965-422a-b814-0a89bc25fc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" Nov 02 20:46:53.081360 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:53.081291 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkcnw\" (UniqueName: \"kubernetes.io/projected/0a94273d-8965-422a-b814-0a89bc25fc5c-kube-api-access-lkcnw\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29\" (UID: \"0a94273d-8965-422a-b814-0a89bc25fc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" Nov 02 20:46:53.117940 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:53.117900 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" Nov 02 20:46:53.623436 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:53.623402 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29"] Nov 02 20:46:53.623639 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:46:53.623611 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a94273d_8965_422a_b814_0a89bc25fc5c.slice/crio-d2dad7c2d958bcbb8f911c778a211bbc0b8c62ca125fefb7476ebdbffc0a5beb WatchSource:0}: Error finding container d2dad7c2d958bcbb8f911c778a211bbc0b8c62ca125fefb7476ebdbffc0a5beb: Status 404 returned error can't find the container with id d2dad7c2d958bcbb8f911c778a211bbc0b8c62ca125fefb7476ebdbffc0a5beb Nov 02 20:46:54.317163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:54.316664 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:54.488041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:54.488001 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" event={"ID":"0a94273d-8965-422a-b814-0a89bc25fc5c","Type":"ContainerStarted","Data":"718491c0ebd7f7a5ea935859d743a1fbb88a1da3399b96b0fc9972aee7c41d41"} Nov 02 20:46:54.488412 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:54.488364 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" event={"ID":"0a94273d-8965-422a-b814-0a89bc25fc5c","Type":"ContainerStarted","Data":"d2dad7c2d958bcbb8f911c778a211bbc0b8c62ca125fefb7476ebdbffc0a5beb"} Nov 02 20:46:54.561923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:54.561893 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:54.612358 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:54.612280 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" podStartSLOduration=2.612261057 podStartE2EDuration="2.612261057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:46:54.610374143 +0000 UTC m=+1942.900030672" watchObservedRunningTime="2026-11-02 20:46:54.612261057 +0000 UTC m=+1942.901917588" Nov 02 20:46:54.803548 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:54.803518 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:55.496461 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:55.496430 4971 generic.go:356] "Generic (PLEG): container finished" podID="0a94273d-8965-422a-b814-0a89bc25fc5c" containerID="718491c0ebd7f7a5ea935859d743a1fbb88a1da3399b96b0fc9972aee7c41d41" exitCode=0 Nov 02 20:46:55.496855 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:55.496567 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" event={"ID":"0a94273d-8965-422a-b814-0a89bc25fc5c","Type":"ContainerDied","Data":"718491c0ebd7f7a5ea935859d743a1fbb88a1da3399b96b0fc9972aee7c41d41"} Nov 02 20:46:56.582622 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:56.582563 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:56.757751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:56.757679 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" Nov 02 20:46:56.883856 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:56.883820 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a94273d-8965-422a-b814-0a89bc25fc5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a94273d-8965-422a-b814-0a89bc25fc5c-maas-api-openshift-ca\") pod \"0a94273d-8965-422a-b814-0a89bc25fc5c\" (UID: \"0a94273d-8965-422a-b814-0a89bc25fc5c\") " Nov 02 20:46:56.884054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:56.883919 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a94273d-8965-422a-b814-0a89bc25fc5c-kube-api-access-lkcnw\" (UniqueName: \"kubernetes.io/projected/0a94273d-8965-422a-b814-0a89bc25fc5c-kube-api-access-lkcnw\") pod \"0a94273d-8965-422a-b814-0a89bc25fc5c\" (UID: \"0a94273d-8965-422a-b814-0a89bc25fc5c\") " Nov 02 20:46:56.884054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:56.884014 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a94273d-8965-422a-b814-0a89bc25fc5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a94273d-8965-422a-b814-0a89bc25fc5c-maas-api-xks-ca\") pod \"0a94273d-8965-422a-b814-0a89bc25fc5c\" (UID: \"0a94273d-8965-422a-b814-0a89bc25fc5c\") " Nov 02 20:46:56.884226 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:56.884203 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a94273d-8965-422a-b814-0a89bc25fc5c-maas-api-openshift-ca" pod "0a94273d-8965-422a-b814-0a89bc25fc5c" (UID: "0a94273d-8965-422a-b814-0a89bc25fc5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:56.884632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:56.884606 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a94273d-8965-422a-b814-0a89bc25fc5c-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:56.886838 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:56.886801 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a94273d-8965-422a-b814-0a89bc25fc5c-kube-api-access-lkcnw" pod "0a94273d-8965-422a-b814-0a89bc25fc5c" (UID: "0a94273d-8965-422a-b814-0a89bc25fc5c"). InnerVolumeSpecName "kube-api-access-lkcnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:56.886994 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:56.886969 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a94273d-8965-422a-b814-0a89bc25fc5c-maas-api-xks-ca" pod "0a94273d-8965-422a-b814-0a89bc25fc5c" (UID: "0a94273d-8965-422a-b814-0a89bc25fc5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:46:56.986764 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:56.986733 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkcnw\" (UniqueName: \"kubernetes.io/projected/0a94273d-8965-422a-b814-0a89bc25fc5c-kube-api-access-lkcnw\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:56.986764 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:56.986765 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a94273d-8965-422a-b814-0a89bc25fc5c-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:57.571110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:57.570975 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:46:57.572471 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:57.572445 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" event={"ID":"0a94273d-8965-422a-b814-0a89bc25fc5c","Type":"ContainerDied","Data":"d2dad7c2d958bcbb8f911c778a211bbc0b8c62ca125fefb7476ebdbffc0a5beb"} Nov 02 20:46:57.572718 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:57.572688 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2dad7c2d958bcbb8f911c778a211bbc0b8c62ca125fefb7476ebdbffc0a5beb" Nov 02 20:46:57.572831 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:57.572539 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29" Nov 02 20:46:57.847635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:46:57.847599 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:01.401593 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:01.401559 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:03.343511 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:47:03.343475 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:47:03.657585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:03.657557 4971 generic.go:356] "Generic (PLEG): container finished" podID="1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" containerID="9833fdf1e58051ac04e9ebac069cb4f3052cb38a7762058a31623d9bc1c83c1f" exitCode=0 Nov 02 20:47:03.657741 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:03.657632 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" event={"ID":"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3","Type":"ContainerDied","Data":"9833fdf1e58051ac04e9ebac069cb4f3052cb38a7762058a31623d9bc1c83c1f"} Nov 02 20:47:03.657986 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:03.657970 4971 scope.go:122] "RemoveContainer" containerID="9833fdf1e58051ac04e9ebac069cb4f3052cb38a7762058a31623d9bc1c83c1f" Nov 02 20:47:03.659366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:03.659334 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:47:03.659447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:03.659403 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:47:03.659447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:03.659433 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:47:03.663556 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:03.663534 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:47:04.465594 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:04.465563 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq"] Nov 02 20:47:04.668529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:04.668492 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" event={"ID":"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3","Type":"ContainerStarted","Data":"1b6c26b1e6aabc1867eeedad8e1db9ce55fee69121f098d49673565952a48cbc"} Nov 02 20:47:04.851102 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:04.850757 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:04.855144 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:04.853795 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:47:04.855444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:04.855421 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 02 20:47:04.855576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:04.855473 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" podUID="1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 02 20:47:05.101416 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:05.101336 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:05.613448 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:05.613398 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:05.853647 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:05.853607 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 02 20:47:05.853849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:05.853686 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" podUID="1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 02 20:47:06.853619 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:06.853586 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 02 20:47:06.854023 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:06.853638 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" podUID="1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 02 20:47:07.148870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:07.148761 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:07.402857 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:07.402758 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:07.550426 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:07.550389 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv"] Nov 02 20:47:07.595006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:07.594974 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0ea576-6pczv"] Nov 02 20:47:07.649090 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:07.649052 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:07.853453 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:07.853425 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 02 20:47:07.853631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:07.853473 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" podUID="1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 02 20:47:07.866805 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:07.866775 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae015af-a1f4-44a1-9fc0-0b551d21ac42" path="/var/lib/kubelet/pods/dae015af-a1f4-44a1-9fc0-0b551d21ac42/volumes" Nov 02 20:47:08.083485 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:08.083454 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:08.853849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:08.853809 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 02 20:47:08.854068 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:08.853877 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" podUID="1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 02 20:47:09.466171 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.466124 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" podUID="7e2f778f-16a7-45fd-912a-fe2098795d7c" containerName="payload-pre-processing" containerID="cri-o://a10c6446e4b5fa79051d5cb7ea1fee720b51f0ebde7e12036d14ef28b6ef8218" gracePeriod=25 Nov 02 20:47:09.716889 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.716645 4971 generic.go:356] "Generic (PLEG): container finished" podID="7e2f778f-16a7-45fd-912a-fe2098795d7c" containerID="a10c6446e4b5fa79051d5cb7ea1fee720b51f0ebde7e12036d14ef28b6ef8218" exitCode=0 Nov 02 20:47:09.717189 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.717073 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" event={"ID":"7e2f778f-16a7-45fd-912a-fe2098795d7c","Type":"ContainerDied","Data":"a10c6446e4b5fa79051d5cb7ea1fee720b51f0ebde7e12036d14ef28b6ef8218"} Nov 02 20:47:09.765025 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.764997 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" Nov 02 20:47:09.857496 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.857454 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2f778f-16a7-45fd-912a-fe2098795d7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2f778f-16a7-45fd-912a-fe2098795d7c-plugins-config-volume\") pod \"7e2f778f-16a7-45fd-912a-fe2098795d7c\" (UID: \"7e2f778f-16a7-45fd-912a-fe2098795d7c\") " Nov 02 20:47:09.857698 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.857535 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2f778f-16a7-45fd-912a-fe2098795d7c-kube-api-access-2tj5n\" (UniqueName: \"kubernetes.io/projected/7e2f778f-16a7-45fd-912a-fe2098795d7c-kube-api-access-2tj5n\") pod \"7e2f778f-16a7-45fd-912a-fe2098795d7c\" (UID: \"7e2f778f-16a7-45fd-912a-fe2098795d7c\") " Nov 02 20:47:09.857934 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.857906 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2f778f-16a7-45fd-912a-fe2098795d7c-plugins-config-volume" pod "7e2f778f-16a7-45fd-912a-fe2098795d7c" (UID: "7e2f778f-16a7-45fd-912a-fe2098795d7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:47:09.859934 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.859908 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2f778f-16a7-45fd-912a-fe2098795d7c-kube-api-access-2tj5n" pod "7e2f778f-16a7-45fd-912a-fe2098795d7c" (UID: "7e2f778f-16a7-45fd-912a-fe2098795d7c"). InnerVolumeSpecName "kube-api-access-2tj5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:09.874317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.874286 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:47:09.875056 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.874918 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:47:09.876617 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.876591 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:47:09.952396 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.952361 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf"] Nov 02 20:47:09.952645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.952598 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" podUID="2bb5da3b-6414-452c-b701-4809b08b5c37" containerName="istio-proxy" containerID="cri-o://1b7c685c0628138bed89bcb7d280b88f042dd45642a89d50b1dfcaf91b1a01bf" gracePeriod=30 Nov 02 20:47:09.958928 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.958900 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tj5n\" (UniqueName: \"kubernetes.io/projected/7e2f778f-16a7-45fd-912a-fe2098795d7c-kube-api-access-2tj5n\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:09.959069 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.958931 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2f778f-16a7-45fd-912a-fe2098795d7c-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:09.984060 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.983994 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc"] Nov 02 20:47:09.984223 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:09.984200 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" podUID="27615c4a-c472-4862-b72d-95dff20b301f" containerName="istio-proxy" containerID="cri-o://1a4dec54642036f975580a53caff9a2d729d01cd3477f96e9890bfb21830ea3f" gracePeriod=30 Nov 02 20:47:10.726030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:10.725987 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" event={"ID":"7e2f778f-16a7-45fd-912a-fe2098795d7c","Type":"ContainerDied","Data":"2c85efcc56c4e3ea7af85986fcee031bb08fe081e267f00d01b1fd7aa7d6b4b3"} Nov 02 20:47:10.726030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:10.726016 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq" Nov 02 20:47:10.726678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:10.726106 4971 scope.go:122] "RemoveContainer" containerID="a10c6446e4b5fa79051d5cb7ea1fee720b51f0ebde7e12036d14ef28b6ef8218" Nov 02 20:47:10.884068 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:10.884042 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq"] Nov 02 20:47:10.887592 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:10.887565 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-039b89-7967c99699-svgjq"] Nov 02 20:47:11.867130 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:11.867096 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e2f778f-16a7-45fd-912a-fe2098795d7c" path="/var/lib/kubelet/pods/7e2f778f-16a7-45fd-912a-fe2098795d7c/volumes" Nov 02 20:47:12.464358 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:12.464297 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:13.990551 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:13.990514 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:14.244920 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:14.244839 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:14.487719 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:14.487687 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:15.104526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.104498 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq"] Nov 02 20:47:15.105529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.105506 4971 cpu_manager.go:414] "containerMap: removing container" podUID="7e2f778f-16a7-45fd-912a-fe2098795d7c" containerName="payload-pre-processing" Nov 02 20:47:15.105676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.105534 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2f778f-16a7-45fd-912a-fe2098795d7c" containerName="payload-pre-processing" Nov 02 20:47:15.105676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.105552 4971 cpu_manager.go:414] "containerMap: removing container" podUID="0a94273d-8965-422a-b814-0a89bc25fc5c" containerName="revoke-keys" Nov 02 20:47:15.105676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.105559 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a94273d-8965-422a-b814-0a89bc25fc5c" containerName="revoke-keys" Nov 02 20:47:15.105910 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.105764 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2f778f-16a7-45fd-912a-fe2098795d7c" containerName="payload-pre-processing" Nov 02 20:47:15.105910 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.105780 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a94273d-8965-422a-b814-0a89bc25fc5c" containerName="revoke-keys" Nov 02 20:47:15.133126 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.132192 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq" Nov 02 20:47:15.181706 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.174882 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq"] Nov 02 20:47:15.209665 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.209624 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:47:15.209842 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.209733 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:47:15.229466 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.229402 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq\" (UID: \"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq" Nov 02 20:47:15.229592 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.229471 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq\" (UID: \"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq" Nov 02 20:47:15.229786 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.229764 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84gjr\" (UniqueName: \"kubernetes.io/projected/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-kube-api-access-84gjr\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq\" (UID: \"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq" Nov 02 20:47:15.293649 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.293625 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:47:15.325958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.325937 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:47:15.331457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.331434 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84gjr\" (UniqueName: \"kubernetes.io/projected/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-kube-api-access-84gjr\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq\" (UID: \"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq" Nov 02 20:47:15.331614 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.331595 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq\" (UID: \"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq" Nov 02 20:47:15.331676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.331662 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq\" (UID: \"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq" Nov 02 20:47:15.332386 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.332322 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq\" (UID: \"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq" Nov 02 20:47:15.333911 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.333888 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq\" (UID: \"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq" Nov 02 20:47:15.389456 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.389432 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84gjr\" (UniqueName: \"kubernetes.io/projected/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-kube-api-access-84gjr\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq\" (UID: \"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq" Nov 02 20:47:15.432607 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.432582 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-istio-envoy\") pod \"27615c4a-c472-4862-b72d-95dff20b301f\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " Nov 02 20:47:15.432769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.432630 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-envoy\") pod \"2bb5da3b-6414-452c-b701-4809b08b5c37\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " Nov 02 20:47:15.432769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.432667 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-workload-certs\") pod \"2bb5da3b-6414-452c-b701-4809b08b5c37\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " Nov 02 20:47:15.432769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.432693 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-token\" (UniqueName: \"kubernetes.io/projected/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-token\") pod \"2bb5da3b-6414-452c-b701-4809b08b5c37\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " Nov 02 20:47:15.432769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.432734 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-workload-certs\") pod \"27615c4a-c472-4862-b72d-95dff20b301f\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " Nov 02 20:47:15.432769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.432760 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-workload-socket\") pod \"27615c4a-c472-4862-b72d-95dff20b301f\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " Nov 02 20:47:15.433070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.432809 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-podinfo\") pod \"2bb5da3b-6414-452c-b701-4809b08b5c37\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " Nov 02 20:47:15.433070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.432851 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb5da3b-6414-452c-b701-4809b08b5c37-kube-api-access-sjbrq\" (UniqueName: \"kubernetes.io/projected/2bb5da3b-6414-452c-b701-4809b08b5c37-kube-api-access-sjbrq\") pod \"2bb5da3b-6414-452c-b701-4809b08b5c37\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " Nov 02 20:47:15.433070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.432901 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27615c4a-c472-4862-b72d-95dff20b301f-istio-token\" (UniqueName: \"kubernetes.io/projected/27615c4a-c472-4862-b72d-95dff20b301f-istio-token\") pod \"27615c4a-c472-4862-b72d-95dff20b301f\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " Nov 02 20:47:15.433070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.432932 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27615c4a-c472-4862-b72d-95dff20b301f-kube-api-access-rh4bs\" (UniqueName: \"kubernetes.io/projected/27615c4a-c472-4862-b72d-95dff20b301f-kube-api-access-rh4bs\") pod \"27615c4a-c472-4862-b72d-95dff20b301f\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " Nov 02 20:47:15.433070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.432946 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-workload-certs" pod "2bb5da3b-6414-452c-b701-4809b08b5c37" (UID: "2bb5da3b-6414-452c-b701-4809b08b5c37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:15.433070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.432960 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-istio-data\") pod \"27615c4a-c472-4862-b72d-95dff20b301f\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " Nov 02 20:47:15.433070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.433030 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27615c4a-c472-4862-b72d-95dff20b301f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27615c4a-c472-4862-b72d-95dff20b301f-istiod-ca-cert\") pod \"27615c4a-c472-4862-b72d-95dff20b301f\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " Nov 02 20:47:15.433537 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.433144 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-workload-socket\") pod \"2bb5da3b-6414-452c-b701-4809b08b5c37\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " Nov 02 20:47:15.433537 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.433214 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-credential-socket\") pod \"27615c4a-c472-4862-b72d-95dff20b301f\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " Nov 02 20:47:15.433537 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.433297 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-credential-socket\") pod \"2bb5da3b-6414-452c-b701-4809b08b5c37\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " Nov 02 20:47:15.433537 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.433352 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-data\") pod \"2bb5da3b-6414-452c-b701-4809b08b5c37\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " Nov 02 20:47:15.433537 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.433447 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27615c4a-c472-4862-b72d-95dff20b301f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27615c4a-c472-4862-b72d-95dff20b301f-istio-podinfo\") pod \"27615c4a-c472-4862-b72d-95dff20b301f\" (UID: \"27615c4a-c472-4862-b72d-95dff20b301f\") " Nov 02 20:47:15.433537 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.433482 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb5da3b-6414-452c-b701-4809b08b5c37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb5da3b-6414-452c-b701-4809b08b5c37-istiod-ca-cert\") pod \"2bb5da3b-6414-452c-b701-4809b08b5c37\" (UID: \"2bb5da3b-6414-452c-b701-4809b08b5c37\") " Nov 02 20:47:15.433871 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.433738 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-istio-data" pod "27615c4a-c472-4862-b72d-95dff20b301f" (UID: "27615c4a-c472-4862-b72d-95dff20b301f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:15.434490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.434067 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-workload-certs" pod "27615c4a-c472-4862-b72d-95dff20b301f" (UID: "27615c4a-c472-4862-b72d-95dff20b301f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:15.434490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.434231 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb5da3b-6414-452c-b701-4809b08b5c37-istiod-ca-cert" pod "2bb5da3b-6414-452c-b701-4809b08b5c37" (UID: "2bb5da3b-6414-452c-b701-4809b08b5c37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:47:15.434490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.434236 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-workload-socket" pod "27615c4a-c472-4862-b72d-95dff20b301f" (UID: "27615c4a-c472-4862-b72d-95dff20b301f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:15.434490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.434305 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-credential-socket" pod "27615c4a-c472-4862-b72d-95dff20b301f" (UID: "27615c4a-c472-4862-b72d-95dff20b301f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:15.434490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.434314 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.434490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.434339 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.434490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.434354 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.434490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.434403 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-credential-socket" pod "2bb5da3b-6414-452c-b701-4809b08b5c37" (UID: "2bb5da3b-6414-452c-b701-4809b08b5c37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:15.434490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.434466 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-data" pod "2bb5da3b-6414-452c-b701-4809b08b5c37" (UID: "2bb5da3b-6414-452c-b701-4809b08b5c37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:15.435110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.434573 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-workload-socket" pod "2bb5da3b-6414-452c-b701-4809b08b5c37" (UID: "2bb5da3b-6414-452c-b701-4809b08b5c37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:15.435110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.434618 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27615c4a-c472-4862-b72d-95dff20b301f-istiod-ca-cert" pod "27615c4a-c472-4862-b72d-95dff20b301f" (UID: "27615c4a-c472-4862-b72d-95dff20b301f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:47:15.436341 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.436181 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-token" pod "2bb5da3b-6414-452c-b701-4809b08b5c37" (UID: "2bb5da3b-6414-452c-b701-4809b08b5c37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:15.436788 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.436754 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27615c4a-c472-4862-b72d-95dff20b301f-kube-api-access-rh4bs" pod "27615c4a-c472-4862-b72d-95dff20b301f" (UID: "27615c4a-c472-4862-b72d-95dff20b301f"). InnerVolumeSpecName "kube-api-access-rh4bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:15.436952 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.436929 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27615c4a-c472-4862-b72d-95dff20b301f-istio-podinfo" pod "27615c4a-c472-4862-b72d-95dff20b301f" (UID: "27615c4a-c472-4862-b72d-95dff20b301f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:47:15.437580 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.437560 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-envoy" pod "2bb5da3b-6414-452c-b701-4809b08b5c37" (UID: "2bb5da3b-6414-452c-b701-4809b08b5c37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:15.437711 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.437687 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-podinfo" pod "2bb5da3b-6414-452c-b701-4809b08b5c37" (UID: "2bb5da3b-6414-452c-b701-4809b08b5c37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:47:15.437767 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.437691 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27615c4a-c472-4862-b72d-95dff20b301f-istio-token" pod "27615c4a-c472-4862-b72d-95dff20b301f" (UID: "27615c4a-c472-4862-b72d-95dff20b301f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:15.438325 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.438301 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb5da3b-6414-452c-b701-4809b08b5c37-kube-api-access-sjbrq" pod "2bb5da3b-6414-452c-b701-4809b08b5c37" (UID: "2bb5da3b-6414-452c-b701-4809b08b5c37"). InnerVolumeSpecName "kube-api-access-sjbrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:15.438434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.438346 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-istio-envoy" pod "27615c4a-c472-4862-b72d-95dff20b301f" (UID: "27615c4a-c472-4862-b72d-95dff20b301f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:15.448594 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.448571 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq" Nov 02 20:47:15.535628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.535590 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.535628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.535621 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27615c4a-c472-4862-b72d-95dff20b301f-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.535628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.535630 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb5da3b-6414-452c-b701-4809b08b5c37-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.535977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.535663 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.535977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.535672 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.535977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.535680 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.535977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.535689 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.535977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.535697 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb5da3b-6414-452c-b701-4809b08b5c37-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.535977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.535706 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjbrq\" (UniqueName: \"kubernetes.io/projected/2bb5da3b-6414-452c-b701-4809b08b5c37-kube-api-access-sjbrq\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.535977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.535715 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27615c4a-c472-4862-b72d-95dff20b301f-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.535977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.535723 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh4bs\" (UniqueName: \"kubernetes.io/projected/27615c4a-c472-4862-b72d-95dff20b301f-kube-api-access-rh4bs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.535977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.535734 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27615c4a-c472-4862-b72d-95dff20b301f-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.535977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.535744 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.535977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.535753 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27615c4a-c472-4862-b72d-95dff20b301f-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.535977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.535762 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb5da3b-6414-452c-b701-4809b08b5c37-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:15.764992 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.764916 4971 generic.go:356] "Generic (PLEG): container finished" podID="27615c4a-c472-4862-b72d-95dff20b301f" containerID="1a4dec54642036f975580a53caff9a2d729d01cd3477f96e9890bfb21830ea3f" exitCode=0 Nov 02 20:47:15.764992 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.764961 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" Nov 02 20:47:15.765257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.764992 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" event={"ID":"27615c4a-c472-4862-b72d-95dff20b301f","Type":"ContainerDied","Data":"1a4dec54642036f975580a53caff9a2d729d01cd3477f96e9890bfb21830ea3f"} Nov 02 20:47:15.765257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.765046 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc" event={"ID":"27615c4a-c472-4862-b72d-95dff20b301f","Type":"ContainerDied","Data":"91dc11f0f74ce805ab9417b29952901a1b92d30002e23fdac771074dc30bacd3"} Nov 02 20:47:15.765257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.765068 4971 scope.go:122] "RemoveContainer" containerID="1a4dec54642036f975580a53caff9a2d729d01cd3477f96e9890bfb21830ea3f" Nov 02 20:47:15.766988 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.766963 4971 generic.go:356] "Generic (PLEG): container finished" podID="2bb5da3b-6414-452c-b701-4809b08b5c37" containerID="1b7c685c0628138bed89bcb7d280b88f042dd45642a89d50b1dfcaf91b1a01bf" exitCode=0 Nov 02 20:47:15.767153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.766998 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" event={"ID":"2bb5da3b-6414-452c-b701-4809b08b5c37","Type":"ContainerDied","Data":"1b7c685c0628138bed89bcb7d280b88f042dd45642a89d50b1dfcaf91b1a01bf"} Nov 02 20:47:15.767153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.767053 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" event={"ID":"2bb5da3b-6414-452c-b701-4809b08b5c37","Type":"ContainerDied","Data":"543748d87dea7070dcba8139650cde55737aaa4825ee59eac1cde965ff555ade"} Nov 02 20:47:15.767153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.767011 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf" Nov 02 20:47:15.776334 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.776312 4971 scope.go:122] "RemoveContainer" containerID="1a4dec54642036f975580a53caff9a2d729d01cd3477f96e9890bfb21830ea3f" Nov 02 20:47:15.776715 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:47:15.776606 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4dec54642036f975580a53caff9a2d729d01cd3477f96e9890bfb21830ea3f\": container with ID starting with 1a4dec54642036f975580a53caff9a2d729d01cd3477f96e9890bfb21830ea3f not found: ID does not exist" containerID="1a4dec54642036f975580a53caff9a2d729d01cd3477f96e9890bfb21830ea3f" Nov 02 20:47:15.776715 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.776640 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4dec54642036f975580a53caff9a2d729d01cd3477f96e9890bfb21830ea3f"} err="failed to get container status \"1a4dec54642036f975580a53caff9a2d729d01cd3477f96e9890bfb21830ea3f\": rpc error: code = NotFound desc = could not find container \"1a4dec54642036f975580a53caff9a2d729d01cd3477f96e9890bfb21830ea3f\": container with ID starting with 1a4dec54642036f975580a53caff9a2d729d01cd3477f96e9890bfb21830ea3f not found: ID does not exist" Nov 02 20:47:15.776715 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.776662 4971 scope.go:122] "RemoveContainer" containerID="1b7c685c0628138bed89bcb7d280b88f042dd45642a89d50b1dfcaf91b1a01bf" Nov 02 20:47:15.789919 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.789903 4971 scope.go:122] "RemoveContainer" containerID="1b7c685c0628138bed89bcb7d280b88f042dd45642a89d50b1dfcaf91b1a01bf" Nov 02 20:47:15.790248 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:47:15.790229 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b7c685c0628138bed89bcb7d280b88f042dd45642a89d50b1dfcaf91b1a01bf\": container with ID starting with 1b7c685c0628138bed89bcb7d280b88f042dd45642a89d50b1dfcaf91b1a01bf not found: ID does not exist" containerID="1b7c685c0628138bed89bcb7d280b88f042dd45642a89d50b1dfcaf91b1a01bf" Nov 02 20:47:15.790366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.790255 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b7c685c0628138bed89bcb7d280b88f042dd45642a89d50b1dfcaf91b1a01bf"} err="failed to get container status \"1b7c685c0628138bed89bcb7d280b88f042dd45642a89d50b1dfcaf91b1a01bf\": rpc error: code = NotFound desc = could not find container \"1b7c685c0628138bed89bcb7d280b88f042dd45642a89d50b1dfcaf91b1a01bf\": container with ID starting with 1b7c685c0628138bed89bcb7d280b88f042dd45642a89d50b1dfcaf91b1a01bf not found: ID does not exist" Nov 02 20:47:15.822006 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:47:15.821981 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49eef5b_35e3_4b9d_acb9_dd7469e76b7f.slice/crio-ab7779c509033a8dc6b73e2346eb6af45c8d86e42cb2bf992e5e7b8316069cb2 WatchSource:0}: Error finding container ab7779c509033a8dc6b73e2346eb6af45c8d86e42cb2bf992e5e7b8316069cb2: Status 404 returned error can't find the container with id ab7779c509033a8dc6b73e2346eb6af45c8d86e42cb2bf992e5e7b8316069cb2 Nov 02 20:47:15.822319 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.822300 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq"] Nov 02 20:47:15.923184 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.923160 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc"] Nov 02 20:47:15.928976 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:15.928956 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-z6cnc"] Nov 02 20:47:16.033816 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:16.033678 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf"] Nov 02 20:47:16.037423 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:16.037402 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-df4jf"] Nov 02 20:47:16.776516 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:16.776488 4971 generic.go:356] "Generic (PLEG): container finished" podID="d49eef5b-35e3-4b9d-acb9-dd7469e76b7f" containerID="a86d68bbb1fa248dde992c9034e55b680112a6e08b240553b6cf4cad9f5c216a" exitCode=0 Nov 02 20:47:16.776861 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:16.776577 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq" event={"ID":"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f","Type":"ContainerDied","Data":"a86d68bbb1fa248dde992c9034e55b680112a6e08b240553b6cf4cad9f5c216a"} Nov 02 20:47:16.776861 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:16.776638 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq" event={"ID":"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f","Type":"ContainerStarted","Data":"ab7779c509033a8dc6b73e2346eb6af45c8d86e42cb2bf992e5e7b8316069cb2"} Nov 02 20:47:17.527945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:17.527913 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:17.872185 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:17.872150 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27615c4a-c472-4862-b72d-95dff20b301f" path="/var/lib/kubelet/pods/27615c4a-c472-4862-b72d-95dff20b301f/volumes" Nov 02 20:47:17.872850 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:17.872828 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb5da3b-6414-452c-b701-4809b08b5c37" path="/var/lib/kubelet/pods/2bb5da3b-6414-452c-b701-4809b08b5c37/volumes" Nov 02 20:47:17.943787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:17.943768 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq" Nov 02 20:47:18.062952 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:18.062883 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-maas-api-openshift-ca\") pod \"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f\" (UID: \"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f\") " Nov 02 20:47:18.062952 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:18.062919 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-kube-api-access-84gjr\" (UniqueName: \"kubernetes.io/projected/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-kube-api-access-84gjr\") pod \"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f\" (UID: \"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f\") " Nov 02 20:47:18.063274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:18.063037 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-maas-api-xks-ca\") pod \"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f\" (UID: \"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f\") " Nov 02 20:47:18.063274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:18.063225 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-maas-api-openshift-ca" pod "d49eef5b-35e3-4b9d-acb9-dd7469e76b7f" (UID: "d49eef5b-35e3-4b9d-acb9-dd7469e76b7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:47:18.063531 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:18.063507 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:18.065277 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:18.065246 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-kube-api-access-84gjr" pod "d49eef5b-35e3-4b9d-acb9-dd7469e76b7f" (UID: "d49eef5b-35e3-4b9d-acb9-dd7469e76b7f"). InnerVolumeSpecName "kube-api-access-84gjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:18.065434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:18.065413 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-maas-api-xks-ca" pod "d49eef5b-35e3-4b9d-acb9-dd7469e76b7f" (UID: "d49eef5b-35e3-4b9d-acb9-dd7469e76b7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:47:18.164509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:18.164484 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84gjr\" (UniqueName: \"kubernetes.io/projected/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-kube-api-access-84gjr\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:18.164509 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:18.164509 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:18.794557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:18.794526 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq" Nov 02 20:47:18.794557 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:18.794537 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq" event={"ID":"d49eef5b-35e3-4b9d-acb9-dd7469e76b7f","Type":"ContainerDied","Data":"ab7779c509033a8dc6b73e2346eb6af45c8d86e42cb2bf992e5e7b8316069cb2"} Nov 02 20:47:18.794837 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:18.794607 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab7779c509033a8dc6b73e2346eb6af45c8d86e42cb2bf992e5e7b8316069cb2" Nov 02 20:47:22.689583 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:22.689548 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:27.848784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:27.848717 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:29.994192 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:29.994149 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:30.247251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:30.245746 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:32.326118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:32.320852 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:33.179533 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:33.179494 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:33.339518 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:47:33.339484 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:47:34.019236 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:34.019193 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:34.268973 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:34.268938 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:34.396227 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:34.396191 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq"] Nov 02 20:47:34.442199 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:34.442157 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-18fd4c-bncmq"] Nov 02 20:47:34.520046 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:34.520015 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:34.952679 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:34.952642 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:35.503246 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:35.503209 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:47:35.503623 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:35.503288 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" podUID="7906ad9c-8297-4513-aaf2-8729d580bfa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:47:35.868006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:35.867973 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d49eef5b-35e3-4b9d-acb9-dd7469e76b7f" path="/var/lib/kubelet/pods/d49eef5b-35e3-4b9d-acb9-dd7469e76b7f/volumes" Nov 02 20:47:36.924797 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:36.924762 4971 generic.go:356] "Generic (PLEG): container finished" podID="7906ad9c-8297-4513-aaf2-8729d580bfa5" containerID="78cc9c5d21c26ab1e80a89b299840e5a902490637285aadc77ece1b75fc15b90" exitCode=0 Nov 02 20:47:36.925266 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:36.924815 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" event={"ID":"7906ad9c-8297-4513-aaf2-8729d580bfa5","Type":"ContainerDied","Data":"78cc9c5d21c26ab1e80a89b299840e5a902490637285aadc77ece1b75fc15b90"} Nov 02 20:47:36.925266 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:36.925154 4971 scope.go:122] "RemoveContainer" containerID="78cc9c5d21c26ab1e80a89b299840e5a902490637285aadc77ece1b75fc15b90" Nov 02 20:47:36.926462 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:36.926436 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:47:36.926521 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:36.926503 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:47:36.926563 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:36.926533 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:47:37.956735 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:37.956699 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" event={"ID":"7906ad9c-8297-4513-aaf2-8729d580bfa5","Type":"ContainerStarted","Data":"f663b599c4f90b1b97518b873901265237ffa18d0fab8ebabfb1f99d46614a2a"} Nov 02 20:47:38.293575 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:38.293475 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:47:38.293751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:38.293655 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 02 20:47:38.293751 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:38.293717 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" podUID="7906ad9c-8297-4513-aaf2-8729d580bfa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 02 20:47:39.294062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:39.294022 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 02 20:47:39.294635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:39.294116 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" podUID="7906ad9c-8297-4513-aaf2-8729d580bfa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 02 20:47:39.347113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:39.346583 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:40.293210 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:40.293175 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 02 20:47:40.293393 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:40.293239 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" podUID="7906ad9c-8297-4513-aaf2-8729d580bfa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 02 20:47:41.086348 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:41.086316 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:41.293368 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:41.293334 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 02 20:47:41.293563 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:41.293392 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" podUID="7906ad9c-8297-4513-aaf2-8729d580bfa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 02 20:47:41.303949 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:41.303923 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc"] Nov 02 20:47:41.304167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:41.304147 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" podUID="b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" containerName="istio-proxy" containerID="cri-o://c272dc93126737a4b0159ee87442b2e92c7bb4b177cc7add914bfbb4134579e3" gracePeriod=30 Nov 02 20:47:41.348899 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:41.348840 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:41.595805 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:41.595771 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:41.894058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:41.894015 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:42.057780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.057742 4971 scope.go:122] "RemoveContainer" containerID="fa6707ae85f40c9eaae5dba551b7f8a0e15b6dbc348e970d562e0f1ad7753f5e" Nov 02 20:47:42.270417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.270335 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9"] Nov 02 20:47:42.271099 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.271064 4971 cpu_manager.go:414] "containerMap: removing container" podUID="d49eef5b-35e3-4b9d-acb9-dd7469e76b7f" containerName="revoke-keys" Nov 02 20:47:42.271099 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.271100 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d49eef5b-35e3-4b9d-acb9-dd7469e76b7f" containerName="revoke-keys" Nov 02 20:47:42.271258 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.271130 4971 cpu_manager.go:414] "containerMap: removing container" podUID="27615c4a-c472-4862-b72d-95dff20b301f" containerName="istio-proxy" Nov 02 20:47:42.271258 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.271136 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27615c4a-c472-4862-b72d-95dff20b301f" containerName="istio-proxy" Nov 02 20:47:42.271258 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.271163 4971 cpu_manager.go:414] "containerMap: removing container" podUID="2bb5da3b-6414-452c-b701-4809b08b5c37" containerName="istio-proxy" Nov 02 20:47:42.271258 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.271168 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb5da3b-6414-452c-b701-4809b08b5c37" containerName="istio-proxy" Nov 02 20:47:42.271391 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.271271 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb5da3b-6414-452c-b701-4809b08b5c37" containerName="istio-proxy" Nov 02 20:47:42.271391 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.271281 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="d49eef5b-35e3-4b9d-acb9-dd7469e76b7f" containerName="revoke-keys" Nov 02 20:47:42.271391 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.271292 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="27615c4a-c472-4862-b72d-95dff20b301f" containerName="istio-proxy" Nov 02 20:47:42.276362 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.276344 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9" Nov 02 20:47:42.293044 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.293020 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 02 20:47:42.293211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.293060 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" podUID="7906ad9c-8297-4513-aaf2-8729d580bfa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 02 20:47:42.351247 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.351217 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:47:42.353625 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.353596 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9"] Nov 02 20:47:42.393774 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.393745 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74rvb\" (UniqueName: \"kubernetes.io/projected/c9063aea-4796-49d4-b7c3-e7ae34292bc8-kube-api-access-74rvb\") pod \"maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9\" (UID: \"c9063aea-4796-49d4-b7c3-e7ae34292bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9" Nov 02 20:47:42.393951 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.393936 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9063aea-4796-49d4-b7c3-e7ae34292bc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9\" (UID: \"c9063aea-4796-49d4-b7c3-e7ae34292bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9" Nov 02 20:47:42.394008 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.393975 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9063aea-4796-49d4-b7c3-e7ae34292bc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9\" (UID: \"c9063aea-4796-49d4-b7c3-e7ae34292bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9" Nov 02 20:47:42.446415 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.446388 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:47:42.495662 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.495636 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9063aea-4796-49d4-b7c3-e7ae34292bc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9\" (UID: \"c9063aea-4796-49d4-b7c3-e7ae34292bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9" Nov 02 20:47:42.495810 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.495683 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9063aea-4796-49d4-b7c3-e7ae34292bc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9\" (UID: \"c9063aea-4796-49d4-b7c3-e7ae34292bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9" Nov 02 20:47:42.495810 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.495756 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74rvb\" (UniqueName: \"kubernetes.io/projected/c9063aea-4796-49d4-b7c3-e7ae34292bc8-kube-api-access-74rvb\") pod \"maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9\" (UID: \"c9063aea-4796-49d4-b7c3-e7ae34292bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9" Nov 02 20:47:42.496449 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.496423 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9063aea-4796-49d4-b7c3-e7ae34292bc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9\" (UID: \"c9063aea-4796-49d4-b7c3-e7ae34292bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9" Nov 02 20:47:42.498225 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.498203 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9063aea-4796-49d4-b7c3-e7ae34292bc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9\" (UID: \"c9063aea-4796-49d4-b7c3-e7ae34292bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9" Nov 02 20:47:42.554030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.553959 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74rvb\" (UniqueName: \"kubernetes.io/projected/c9063aea-4796-49d4-b7c3-e7ae34292bc8-kube-api-access-74rvb\") pod \"maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9\" (UID: \"c9063aea-4796-49d4-b7c3-e7ae34292bc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9" Nov 02 20:47:42.590440 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.590416 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9" Nov 02 20:47:42.992260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:42.988970 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9"] Nov 02 20:47:43.001413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:43.001387 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9" event={"ID":"c9063aea-4796-49d4-b7c3-e7ae34292bc8","Type":"ContainerStarted","Data":"6fa8254a485910e7ca747c7114b4025771741188722e4bc97bdd5978b8ab06dc"} Nov 02 20:47:43.075891 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:43.075854 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:43.297040 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:43.297012 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:47:43.297559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:43.297531 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:47:43.298427 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:43.298409 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:47:44.010959 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:44.010927 4971 generic.go:356] "Generic (PLEG): container finished" podID="c9063aea-4796-49d4-b7c3-e7ae34292bc8" containerID="1a9e7ada215e404d59586e092643db0be90008d29fa8fd469f8c44953e4fba50" exitCode=0 Nov 02 20:47:44.011124 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:44.011010 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9" event={"ID":"c9063aea-4796-49d4-b7c3-e7ae34292bc8","Type":"ContainerDied","Data":"1a9e7ada215e404d59586e092643db0be90008d29fa8fd469f8c44953e4fba50"} Nov 02 20:47:44.142677 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:44.142650 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t"] Nov 02 20:47:44.147017 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:44.146966 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" Nov 02 20:47:44.236978 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:44.236948 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t"] Nov 02 20:47:44.319334 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:44.319302 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5305b6-a8a8-4a7a-854b-233591f338d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t\" (UID: \"6a5305b6-a8a8-4a7a-854b-233591f338d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" Nov 02 20:47:44.319765 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:44.319425 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55fmc\" (UniqueName: \"kubernetes.io/projected/6a5305b6-a8a8-4a7a-854b-233591f338d0-kube-api-access-55fmc\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t\" (UID: \"6a5305b6-a8a8-4a7a-854b-233591f338d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" Nov 02 20:47:44.319765 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:44.319554 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5305b6-a8a8-4a7a-854b-233591f338d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t\" (UID: \"6a5305b6-a8a8-4a7a-854b-233591f338d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" Nov 02 20:47:44.420257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:44.420181 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5305b6-a8a8-4a7a-854b-233591f338d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t\" (UID: \"6a5305b6-a8a8-4a7a-854b-233591f338d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" Nov 02 20:47:44.420413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:44.420256 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5305b6-a8a8-4a7a-854b-233591f338d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t\" (UID: \"6a5305b6-a8a8-4a7a-854b-233591f338d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" Nov 02 20:47:44.420413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:44.420334 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55fmc\" (UniqueName: \"kubernetes.io/projected/6a5305b6-a8a8-4a7a-854b-233591f338d0-kube-api-access-55fmc\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t\" (UID: \"6a5305b6-a8a8-4a7a-854b-233591f338d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" Nov 02 20:47:44.421005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:44.420984 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5305b6-a8a8-4a7a-854b-233591f338d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t\" (UID: \"6a5305b6-a8a8-4a7a-854b-233591f338d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" Nov 02 20:47:44.422832 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:44.422802 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5305b6-a8a8-4a7a-854b-233591f338d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t\" (UID: \"6a5305b6-a8a8-4a7a-854b-233591f338d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" Nov 02 20:47:44.478108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:44.478071 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55fmc\" (UniqueName: \"kubernetes.io/projected/6a5305b6-a8a8-4a7a-854b-233591f338d0-kube-api-access-55fmc\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t\" (UID: \"6a5305b6-a8a8-4a7a-854b-233591f338d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" Nov 02 20:47:44.566306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:44.566277 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:44.760917 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:44.760737 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" Nov 02 20:47:45.138692 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:45.138667 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t"] Nov 02 20:47:45.267685 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:45.267665 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9" Nov 02 20:47:45.331912 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:45.331861 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9063aea-4796-49d4-b7c3-e7ae34292bc8-kube-api-access-74rvb\" (UniqueName: \"kubernetes.io/projected/c9063aea-4796-49d4-b7c3-e7ae34292bc8-kube-api-access-74rvb\") pod \"c9063aea-4796-49d4-b7c3-e7ae34292bc8\" (UID: \"c9063aea-4796-49d4-b7c3-e7ae34292bc8\") " Nov 02 20:47:45.332315 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:45.331967 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9063aea-4796-49d4-b7c3-e7ae34292bc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9063aea-4796-49d4-b7c3-e7ae34292bc8-maas-api-openshift-ca\") pod \"c9063aea-4796-49d4-b7c3-e7ae34292bc8\" (UID: \"c9063aea-4796-49d4-b7c3-e7ae34292bc8\") " Nov 02 20:47:45.332315 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:45.332103 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9063aea-4796-49d4-b7c3-e7ae34292bc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9063aea-4796-49d4-b7c3-e7ae34292bc8-maas-api-xks-ca\") pod \"c9063aea-4796-49d4-b7c3-e7ae34292bc8\" (UID: \"c9063aea-4796-49d4-b7c3-e7ae34292bc8\") " Nov 02 20:47:45.332446 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:45.332354 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9063aea-4796-49d4-b7c3-e7ae34292bc8-maas-api-openshift-ca" pod "c9063aea-4796-49d4-b7c3-e7ae34292bc8" (UID: "c9063aea-4796-49d4-b7c3-e7ae34292bc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:47:45.332524 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:45.332507 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9063aea-4796-49d4-b7c3-e7ae34292bc8-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:45.334251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:45.334231 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9063aea-4796-49d4-b7c3-e7ae34292bc8-kube-api-access-74rvb" pod "c9063aea-4796-49d4-b7c3-e7ae34292bc8" (UID: "c9063aea-4796-49d4-b7c3-e7ae34292bc8"). InnerVolumeSpecName "kube-api-access-74rvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:45.334337 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:45.334265 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9063aea-4796-49d4-b7c3-e7ae34292bc8-maas-api-xks-ca" pod "c9063aea-4796-49d4-b7c3-e7ae34292bc8" (UID: "c9063aea-4796-49d4-b7c3-e7ae34292bc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:47:45.433612 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:45.433575 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9063aea-4796-49d4-b7c3-e7ae34292bc8-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:45.433612 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:45.433612 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74rvb\" (UniqueName: \"kubernetes.io/projected/c9063aea-4796-49d4-b7c3-e7ae34292bc8-kube-api-access-74rvb\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:45.792474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:45.792445 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:47:45.792600 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:45.792498 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" podUID="b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:47:46.028150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.028108 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" event={"ID":"6a5305b6-a8a8-4a7a-854b-233591f338d0","Type":"ContainerStarted","Data":"280f0a44643c9ee9e319a68fe98d0f556484a064c9cac60dabfc20c042f5be9e"} Nov 02 20:47:46.028310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.028185 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" event={"ID":"6a5305b6-a8a8-4a7a-854b-233591f338d0","Type":"ContainerStarted","Data":"c9f42072e1faa5cf5abe6d545020114000a841cc7ff0c57bf83a9f7227177139"} Nov 02 20:47:46.029884 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.029857 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9" event={"ID":"c9063aea-4796-49d4-b7c3-e7ae34292bc8","Type":"ContainerDied","Data":"6fa8254a485910e7ca747c7114b4025771741188722e4bc97bdd5978b8ab06dc"} Nov 02 20:47:46.030017 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.029907 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fa8254a485910e7ca747c7114b4025771741188722e4bc97bdd5978b8ab06dc" Nov 02 20:47:46.030017 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.029911 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9" Nov 02 20:47:46.138191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.138143 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" podStartSLOduration=2.138130215 podStartE2EDuration="2.138130215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:47:46.13685368 +0000 UTC m=+1994.426510209" watchObservedRunningTime="2026-11-02 20:47:46.138130215 +0000 UTC m=+1994.427786734" Nov 02 20:47:46.581243 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.581221 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:47:46.646896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.646869 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-workload-certs\") pod \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " Nov 02 20:47:46.647032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.646901 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istiod-ca-cert\") pod \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " Nov 02 20:47:46.647032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.646962 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-credential-socket\") pod \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " Nov 02 20:47:46.647032 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.646999 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-envoy\") pod \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " Nov 02 20:47:46.647210 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.647061 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-workload-socket\") pod \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " Nov 02 20:47:46.647210 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.647121 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-podinfo\") pod \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " Nov 02 20:47:46.647210 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.647147 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-kube-api-access-k5fqt\" (UniqueName: \"kubernetes.io/projected/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-kube-api-access-k5fqt\") pod \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " Nov 02 20:47:46.647210 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.647165 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-workload-certs" pod "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" (UID: "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:46.647433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.647172 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-token\" (UniqueName: \"kubernetes.io/projected/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-token\") pod \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " Nov 02 20:47:46.647433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.647267 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-data\") pod \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\" (UID: \"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5\") " Nov 02 20:47:46.647433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.647269 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-credential-socket" pod "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" (UID: "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:46.647651 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.647468 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-workload-socket" pod "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" (UID: "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:46.647651 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.647484 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istiod-ca-cert" pod "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" (UID: "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:47:46.647749 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.647715 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-data" pod "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" (UID: "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:46.648063 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.648040 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:46.648162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.648070 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:46.648162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.648103 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:46.648162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.648117 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:46.648162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.648130 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:46.649511 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.649487 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-token" pod "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" (UID: "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:46.649630 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.649611 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-kube-api-access-k5fqt" pod "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" (UID: "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5"). InnerVolumeSpecName "kube-api-access-k5fqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:46.650028 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.650012 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-envoy" pod "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" (UID: "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:46.650028 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.650019 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-podinfo" pod "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" (UID: "b089fd4e-b4a4-4420-b7f5-f8f79b78bba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:47:46.748075 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.748001 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s"] Nov 02 20:47:46.749033 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.749008 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:46.749033 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.749033 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5fqt\" (UniqueName: \"kubernetes.io/projected/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-kube-api-access-k5fqt\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:46.749267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.749044 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:46.749267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.749055 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:46.762414 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:46.762388 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb"] Nov 02 20:47:47.008411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:47.008337 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:47.039122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:47.039076 4971 generic.go:356] "Generic (PLEG): container finished" podID="6a5305b6-a8a8-4a7a-854b-233591f338d0" containerID="280f0a44643c9ee9e319a68fe98d0f556484a064c9cac60dabfc20c042f5be9e" exitCode=0 Nov 02 20:47:47.039122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:47.039110 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" event={"ID":"6a5305b6-a8a8-4a7a-854b-233591f338d0","Type":"ContainerDied","Data":"280f0a44643c9ee9e319a68fe98d0f556484a064c9cac60dabfc20c042f5be9e"} Nov 02 20:47:47.040967 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:47.040946 4971 generic.go:356] "Generic (PLEG): container finished" podID="b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" containerID="c272dc93126737a4b0159ee87442b2e92c7bb4b177cc7add914bfbb4134579e3" exitCode=0 Nov 02 20:47:47.041122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:47.040974 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" event={"ID":"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5","Type":"ContainerDied","Data":"c272dc93126737a4b0159ee87442b2e92c7bb4b177cc7add914bfbb4134579e3"} Nov 02 20:47:47.041122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:47.040993 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" Nov 02 20:47:47.041122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:47.041018 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc" event={"ID":"b089fd4e-b4a4-4420-b7f5-f8f79b78bba5","Type":"ContainerDied","Data":"92142f6e8f66a1213c4a32bd57d2293b83572af456ea9a99b33a415120e81746"} Nov 02 20:47:47.041122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:47.041041 4971 scope.go:122] "RemoveContainer" containerID="c272dc93126737a4b0159ee87442b2e92c7bb4b177cc7add914bfbb4134579e3" Nov 02 20:47:47.051734 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:47.051716 4971 scope.go:122] "RemoveContainer" containerID="eb1298232bf22e960593cd92a2b66778bf1f24c8f598b3d7eb019b1b644261dd" Nov 02 20:47:47.061273 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:47.061258 4971 scope.go:122] "RemoveContainer" containerID="c272dc93126737a4b0159ee87442b2e92c7bb4b177cc7add914bfbb4134579e3" Nov 02 20:47:47.061525 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:47:47.061510 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c272dc93126737a4b0159ee87442b2e92c7bb4b177cc7add914bfbb4134579e3\": container with ID starting with c272dc93126737a4b0159ee87442b2e92c7bb4b177cc7add914bfbb4134579e3 not found: ID does not exist" containerID="c272dc93126737a4b0159ee87442b2e92c7bb4b177cc7add914bfbb4134579e3" Nov 02 20:47:47.061568 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:47.061533 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c272dc93126737a4b0159ee87442b2e92c7bb4b177cc7add914bfbb4134579e3"} err="failed to get container status \"c272dc93126737a4b0159ee87442b2e92c7bb4b177cc7add914bfbb4134579e3\": rpc error: code = NotFound desc = could not find container \"c272dc93126737a4b0159ee87442b2e92c7bb4b177cc7add914bfbb4134579e3\": container with ID starting with c272dc93126737a4b0159ee87442b2e92c7bb4b177cc7add914bfbb4134579e3 not found: ID does not exist" Nov 02 20:47:47.061568 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:47.061548 4971 scope.go:122] "RemoveContainer" containerID="eb1298232bf22e960593cd92a2b66778bf1f24c8f598b3d7eb019b1b644261dd" Nov 02 20:47:47.061759 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:47:47.061743 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1298232bf22e960593cd92a2b66778bf1f24c8f598b3d7eb019b1b644261dd\": container with ID starting with eb1298232bf22e960593cd92a2b66778bf1f24c8f598b3d7eb019b1b644261dd not found: ID does not exist" containerID="eb1298232bf22e960593cd92a2b66778bf1f24c8f598b3d7eb019b1b644261dd" Nov 02 20:47:47.061797 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:47.061765 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1298232bf22e960593cd92a2b66778bf1f24c8f598b3d7eb019b1b644261dd"} err="failed to get container status \"eb1298232bf22e960593cd92a2b66778bf1f24c8f598b3d7eb019b1b644261dd\": rpc error: code = NotFound desc = could not find container \"eb1298232bf22e960593cd92a2b66778bf1f24c8f598b3d7eb019b1b644261dd\": container with ID starting with eb1298232bf22e960593cd92a2b66778bf1f24c8f598b3d7eb019b1b644261dd not found: ID does not exist" Nov 02 20:47:47.256266 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:47.256234 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc"] Nov 02 20:47:47.261675 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:47.261592 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-18fd4c-openshift-default-667875bcb9-xglvc"] Nov 02 20:47:47.868533 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:47.868490 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" path="/var/lib/kubelet/pods/b089fd4e-b4a4-4420-b7f5-f8f79b78bba5/volumes" Nov 02 20:47:48.207844 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:48.207821 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" Nov 02 20:47:48.265624 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:48.265596 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a5305b6-a8a8-4a7a-854b-233591f338d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5305b6-a8a8-4a7a-854b-233591f338d0-maas-api-xks-ca\") pod \"6a5305b6-a8a8-4a7a-854b-233591f338d0\" (UID: \"6a5305b6-a8a8-4a7a-854b-233591f338d0\") " Nov 02 20:47:48.265774 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:48.265719 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5305b6-a8a8-4a7a-854b-233591f338d0-kube-api-access-55fmc\" (UniqueName: \"kubernetes.io/projected/6a5305b6-a8a8-4a7a-854b-233591f338d0-kube-api-access-55fmc\") pod \"6a5305b6-a8a8-4a7a-854b-233591f338d0\" (UID: \"6a5305b6-a8a8-4a7a-854b-233591f338d0\") " Nov 02 20:47:48.265849 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:48.265832 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5305b6-a8a8-4a7a-854b-233591f338d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5305b6-a8a8-4a7a-854b-233591f338d0-maas-api-openshift-ca\") pod \"6a5305b6-a8a8-4a7a-854b-233591f338d0\" (UID: \"6a5305b6-a8a8-4a7a-854b-233591f338d0\") " Nov 02 20:47:48.266193 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:48.266162 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5305b6-a8a8-4a7a-854b-233591f338d0-maas-api-openshift-ca" pod "6a5305b6-a8a8-4a7a-854b-233591f338d0" (UID: "6a5305b6-a8a8-4a7a-854b-233591f338d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:47:48.266320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:48.266305 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5305b6-a8a8-4a7a-854b-233591f338d0-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:48.267737 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:48.267717 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5305b6-a8a8-4a7a-854b-233591f338d0-maas-api-xks-ca" pod "6a5305b6-a8a8-4a7a-854b-233591f338d0" (UID: "6a5305b6-a8a8-4a7a-854b-233591f338d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:47:48.267986 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:48.267906 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5305b6-a8a8-4a7a-854b-233591f338d0-kube-api-access-55fmc" pod "6a5305b6-a8a8-4a7a-854b-233591f338d0" (UID: "6a5305b6-a8a8-4a7a-854b-233591f338d0"). InnerVolumeSpecName "kube-api-access-55fmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:48.367861 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:48.367831 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55fmc\" (UniqueName: \"kubernetes.io/projected/6a5305b6-a8a8-4a7a-854b-233591f338d0-kube-api-access-55fmc\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:48.367861 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:48.367859 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5305b6-a8a8-4a7a-854b-233591f338d0-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:48.980268 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:48.980194 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:49.057476 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.057446 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" Nov 02 20:47:49.057476 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.057465 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t" event={"ID":"6a5305b6-a8a8-4a7a-854b-233591f338d0","Type":"ContainerDied","Data":"c9f42072e1faa5cf5abe6d545020114000a841cc7ff0c57bf83a9f7227177139"} Nov 02 20:47:49.057687 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.057517 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9f42072e1faa5cf5abe6d545020114000a841cc7ff0c57bf83a9f7227177139" Nov 02 20:47:49.301206 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.301076 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885"] Nov 02 20:47:49.302543 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.302455 4971 cpu_manager.go:414] "containerMap: removing container" podUID="b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" containerName="istio-proxy" Nov 02 20:47:49.302543 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.302485 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" containerName="istio-proxy" Nov 02 20:47:49.302543 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.302506 4971 cpu_manager.go:414] "containerMap: removing container" podUID="6a5305b6-a8a8-4a7a-854b-233591f338d0" containerName="revoke-keys" Nov 02 20:47:49.302543 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.302514 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5305b6-a8a8-4a7a-854b-233591f338d0" containerName="revoke-keys" Nov 02 20:47:49.302948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.302562 4971 cpu_manager.go:414] "containerMap: removing container" podUID="b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" containerName="istio-proxy" Nov 02 20:47:49.302948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.302570 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" containerName="istio-proxy" Nov 02 20:47:49.302948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.302600 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c9063aea-4796-49d4-b7c3-e7ae34292bc8" containerName="revoke-keys" Nov 02 20:47:49.302948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.302608 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9063aea-4796-49d4-b7c3-e7ae34292bc8" containerName="revoke-keys" Nov 02 20:47:49.302948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.302745 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" containerName="istio-proxy" Nov 02 20:47:49.302948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.302760 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5305b6-a8a8-4a7a-854b-233591f338d0" containerName="revoke-keys" Nov 02 20:47:49.302948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.302884 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9063aea-4796-49d4-b7c3-e7ae34292bc8" containerName="revoke-keys" Nov 02 20:47:49.307314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.307289 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885" Nov 02 20:47:49.360411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.360384 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:47:49.360584 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.360560 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:47:49.362795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.362769 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885"] Nov 02 20:47:49.386656 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.386629 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2f082b-62a4-4bde-80b8-399f286abd57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885\" (UID: \"2d2f082b-62a4-4bde-80b8-399f286abd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885" Nov 02 20:47:49.386931 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.386890 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kjlg\" (UniqueName: \"kubernetes.io/projected/2d2f082b-62a4-4bde-80b8-399f286abd57-kube-api-access-6kjlg\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885\" (UID: \"2d2f082b-62a4-4bde-80b8-399f286abd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885" Nov 02 20:47:49.387109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.387070 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2f082b-62a4-4bde-80b8-399f286abd57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885\" (UID: \"2d2f082b-62a4-4bde-80b8-399f286abd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885" Nov 02 20:47:49.488685 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.488650 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2f082b-62a4-4bde-80b8-399f286abd57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885\" (UID: \"2d2f082b-62a4-4bde-80b8-399f286abd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885" Nov 02 20:47:49.488814 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.488782 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kjlg\" (UniqueName: \"kubernetes.io/projected/2d2f082b-62a4-4bde-80b8-399f286abd57-kube-api-access-6kjlg\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885\" (UID: \"2d2f082b-62a4-4bde-80b8-399f286abd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885" Nov 02 20:47:49.488861 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.488846 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2f082b-62a4-4bde-80b8-399f286abd57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885\" (UID: \"2d2f082b-62a4-4bde-80b8-399f286abd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885" Nov 02 20:47:49.489516 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.489498 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2f082b-62a4-4bde-80b8-399f286abd57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885\" (UID: \"2d2f082b-62a4-4bde-80b8-399f286abd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885" Nov 02 20:47:49.491177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.491160 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2f082b-62a4-4bde-80b8-399f286abd57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885\" (UID: \"2d2f082b-62a4-4bde-80b8-399f286abd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885" Nov 02 20:47:49.546653 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.546621 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kjlg\" (UniqueName: \"kubernetes.io/projected/2d2f082b-62a4-4bde-80b8-399f286abd57-kube-api-access-6kjlg\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885\" (UID: \"2d2f082b-62a4-4bde-80b8-399f286abd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885" Nov 02 20:47:49.625870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.625849 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885" Nov 02 20:47:49.998722 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:49.998689 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885"] Nov 02 20:47:49.999202 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:47:49.998702 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2f082b_62a4_4bde_80b8_399f286abd57.slice/crio-c29e7953d7cced9898b25c5ea90dfb10f16292229ba3ed1f8d1d819dad2010f8 WatchSource:0}: Error finding container c29e7953d7cced9898b25c5ea90dfb10f16292229ba3ed1f8d1d819dad2010f8: Status 404 returned error can't find the container with id c29e7953d7cced9898b25c5ea90dfb10f16292229ba3ed1f8d1d819dad2010f8 Nov 02 20:47:50.065738 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:50.065707 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885" event={"ID":"2d2f082b-62a4-4bde-80b8-399f286abd57","Type":"ContainerStarted","Data":"c29e7953d7cced9898b25c5ea90dfb10f16292229ba3ed1f8d1d819dad2010f8"} Nov 02 20:47:50.590676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:50.590604 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:50.862379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:50.862340 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:50.984048 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:50.983942 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29"] Nov 02 20:47:51.031177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:51.031139 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-039b89-bzj29"] Nov 02 20:47:51.073777 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:51.073737 4971 generic.go:356] "Generic (PLEG): container finished" podID="2d2f082b-62a4-4bde-80b8-399f286abd57" containerID="cbbea10a947e3dc550dbad7da20d5ad74e9324ec7c1ab63285aed6439512c121" exitCode=0 Nov 02 20:47:51.073905 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:51.073820 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885" event={"ID":"2d2f082b-62a4-4bde-80b8-399f286abd57","Type":"ContainerDied","Data":"cbbea10a947e3dc550dbad7da20d5ad74e9324ec7c1ab63285aed6439512c121"} Nov 02 20:47:51.107944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:51.107923 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:51.474488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:51.474459 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:51.867801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:51.867771 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a94273d-8965-422a-b814-0a89bc25fc5c" path="/var/lib/kubelet/pods/0a94273d-8965-422a-b814-0a89bc25fc5c/volumes" Nov 02 20:47:52.241985 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:52.241959 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885" Nov 02 20:47:52.334316 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:52.334287 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2f082b-62a4-4bde-80b8-399f286abd57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2f082b-62a4-4bde-80b8-399f286abd57-maas-api-openshift-ca\") pod \"2d2f082b-62a4-4bde-80b8-399f286abd57\" (UID: \"2d2f082b-62a4-4bde-80b8-399f286abd57\") " Nov 02 20:47:52.334457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:52.334443 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2f082b-62a4-4bde-80b8-399f286abd57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2f082b-62a4-4bde-80b8-399f286abd57-maas-api-xks-ca\") pod \"2d2f082b-62a4-4bde-80b8-399f286abd57\" (UID: \"2d2f082b-62a4-4bde-80b8-399f286abd57\") " Nov 02 20:47:52.334529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:52.334510 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2f082b-62a4-4bde-80b8-399f286abd57-kube-api-access-6kjlg\" (UniqueName: \"kubernetes.io/projected/2d2f082b-62a4-4bde-80b8-399f286abd57-kube-api-access-6kjlg\") pod \"2d2f082b-62a4-4bde-80b8-399f286abd57\" (UID: \"2d2f082b-62a4-4bde-80b8-399f286abd57\") " Nov 02 20:47:52.334604 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:52.334584 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2f082b-62a4-4bde-80b8-399f286abd57-maas-api-openshift-ca" pod "2d2f082b-62a4-4bde-80b8-399f286abd57" (UID: "2d2f082b-62a4-4bde-80b8-399f286abd57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:47:52.336970 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:52.335640 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2f082b-62a4-4bde-80b8-399f286abd57-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:52.345460 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:52.345433 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2f082b-62a4-4bde-80b8-399f286abd57-maas-api-xks-ca" pod "2d2f082b-62a4-4bde-80b8-399f286abd57" (UID: "2d2f082b-62a4-4bde-80b8-399f286abd57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:47:52.345732 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:52.345460 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2f082b-62a4-4bde-80b8-399f286abd57-kube-api-access-6kjlg" pod "2d2f082b-62a4-4bde-80b8-399f286abd57" (UID: "2d2f082b-62a4-4bde-80b8-399f286abd57"). InnerVolumeSpecName "kube-api-access-6kjlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:52.436479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:52.436410 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2f082b-62a4-4bde-80b8-399f286abd57-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:52.436479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:52.436441 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kjlg\" (UniqueName: \"kubernetes.io/projected/2d2f082b-62a4-4bde-80b8-399f286abd57-kube-api-access-6kjlg\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:52.980925 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:52.980893 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs"] Nov 02 20:47:53.095934 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:53.095892 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885" event={"ID":"2d2f082b-62a4-4bde-80b8-399f286abd57","Type":"ContainerDied","Data":"c29e7953d7cced9898b25c5ea90dfb10f16292229ba3ed1f8d1d819dad2010f8"} Nov 02 20:47:53.096225 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:53.095963 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c29e7953d7cced9898b25c5ea90dfb10f16292229ba3ed1f8d1d819dad2010f8" Nov 02 20:47:53.096225 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:53.096043 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885" Nov 02 20:47:53.313979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:53.313894 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:53.555447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:53.555409 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:55.936476 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:55.936434 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:56.185217 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:56.185181 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:56.461234 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:56.461203 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:57.638047 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:57.638008 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:57.866888 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:57.866860 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6"] Nov 02 20:47:57.867074 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:57.867053 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" podUID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" containerName="istio-proxy" containerID="cri-o://ae4a11cdd15d312e43afd1071dc944a972e76732dac3279b4a7569afea805341" gracePeriod=30 Nov 02 20:47:57.882693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:57.882627 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:57.981580 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:57.981455 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" podUID="5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4" containerName="payload-pre-processing" containerID="cri-o://3ff6e63ca94b047be44ad0a31a8f5c4c01d23d64769a882a6c104efdb78c184e" gracePeriod=25 Nov 02 20:47:58.122734 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:58.122700 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:58.442024 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:58.441992 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:47:58.808178 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:58.808097 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k"] Nov 02 20:47:58.808764 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:58.808738 4971 cpu_manager.go:414] "containerMap: removing container" podUID="2d2f082b-62a4-4bde-80b8-399f286abd57" containerName="revoke-keys" Nov 02 20:47:58.808764 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:58.808759 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2f082b-62a4-4bde-80b8-399f286abd57" containerName="revoke-keys" Nov 02 20:47:58.808918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:58.808876 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="b089fd4e-b4a4-4420-b7f5-f8f79b78bba5" containerName="istio-proxy" Nov 02 20:47:58.808918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:58.808889 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2f082b-62a4-4bde-80b8-399f286abd57" containerName="revoke-keys" Nov 02 20:47:58.812262 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:58.812246 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" Nov 02 20:47:58.865421 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:58.865394 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:47:58.865421 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:58.865409 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:47:58.915922 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:58.915884 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k"] Nov 02 20:47:58.918207 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:58.918186 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9895c4c3-2e66-47a8-9d92-8172c289204b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k\" (UID: \"9895c4c3-2e66-47a8-9d92-8172c289204b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" Nov 02 20:47:58.918358 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:58.918341 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4phbb\" (UniqueName: \"kubernetes.io/projected/9895c4c3-2e66-47a8-9d92-8172c289204b-kube-api-access-4phbb\") pod \"maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k\" (UID: \"9895c4c3-2e66-47a8-9d92-8172c289204b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" Nov 02 20:47:58.918405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:58.918389 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9895c4c3-2e66-47a8-9d92-8172c289204b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k\" (UID: \"9895c4c3-2e66-47a8-9d92-8172c289204b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" Nov 02 20:47:59.019840 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:59.019813 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9895c4c3-2e66-47a8-9d92-8172c289204b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k\" (UID: \"9895c4c3-2e66-47a8-9d92-8172c289204b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" Nov 02 20:47:59.019999 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:59.019903 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9895c4c3-2e66-47a8-9d92-8172c289204b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k\" (UID: \"9895c4c3-2e66-47a8-9d92-8172c289204b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" Nov 02 20:47:59.019999 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:59.019972 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4phbb\" (UniqueName: \"kubernetes.io/projected/9895c4c3-2e66-47a8-9d92-8172c289204b-kube-api-access-4phbb\") pod \"maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k\" (UID: \"9895c4c3-2e66-47a8-9d92-8172c289204b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" Nov 02 20:47:59.020591 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:59.020569 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9895c4c3-2e66-47a8-9d92-8172c289204b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k\" (UID: \"9895c4c3-2e66-47a8-9d92-8172c289204b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" Nov 02 20:47:59.022426 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:59.022403 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9895c4c3-2e66-47a8-9d92-8172c289204b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k\" (UID: \"9895c4c3-2e66-47a8-9d92-8172c289204b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" Nov 02 20:47:59.077713 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:59.077662 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4phbb\" (UniqueName: \"kubernetes.io/projected/9895c4c3-2e66-47a8-9d92-8172c289204b-kube-api-access-4phbb\") pod \"maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k\" (UID: \"9895c4c3-2e66-47a8-9d92-8172c289204b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" Nov 02 20:47:59.125936 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:59.125916 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" Nov 02 20:47:59.498734 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:47:59.498684 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9895c4c3_2e66_47a8_9d92_8172c289204b.slice/crio-3829121501bf22e22f416f79b583784f7103d33690d1885af02591471b492742 WatchSource:0}: Error finding container 3829121501bf22e22f416f79b583784f7103d33690d1885af02591471b492742: Status 404 returned error can't find the container with id 3829121501bf22e22f416f79b583784f7103d33690d1885af02591471b492742 Nov 02 20:47:59.499432 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:59.499270 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k"] Nov 02 20:47:59.920033 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:59.920006 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:47:59.920395 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:47:59.920051 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" podUID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:48:00.146765 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:00.146727 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" event={"ID":"9895c4c3-2e66-47a8-9d92-8172c289204b","Type":"ContainerStarted","Data":"ef03d21af285033422f9a421fab455900c496d3bf1cf66f59a846e85760e204f"} Nov 02 20:48:00.146955 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:00.146784 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" event={"ID":"9895c4c3-2e66-47a8-9d92-8172c289204b","Type":"ContainerStarted","Data":"3829121501bf22e22f416f79b583784f7103d33690d1885af02591471b492742"} Nov 02 20:48:00.256675 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:00.256628 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" podStartSLOduration=2.256613273 podStartE2EDuration="2.256613273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:48:00.256058936 +0000 UTC m=+2008.545715463" watchObservedRunningTime="2026-11-02 20:48:00.256613273 +0000 UTC m=+2008.546269818" Nov 02 20:48:00.385451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:00.385420 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:00.630787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:00.630755 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:01.154711 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:01.154668 4971 generic.go:356] "Generic (PLEG): container finished" podID="9895c4c3-2e66-47a8-9d92-8172c289204b" containerID="ef03d21af285033422f9a421fab455900c496d3bf1cf66f59a846e85760e204f" exitCode=0 Nov 02 20:48:01.155200 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:01.154758 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" event={"ID":"9895c4c3-2e66-47a8-9d92-8172c289204b","Type":"ContainerDied","Data":"ef03d21af285033422f9a421fab455900c496d3bf1cf66f59a846e85760e204f"} Nov 02 20:48:01.515630 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:01.515500 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:01.770435 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:01.770321 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" containerID="cri-o://6c335aac4e8ab3d8c80453327d200e9734b824b441b0f9ab1eaef85540a20e30" gracePeriod=45 Nov 02 20:48:01.775519 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:01.775479 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": EOF" Nov 02 20:48:01.783579 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:01.783545 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" containerID="cri-o://69862c9a561a26a343e043387988201f1889fd6d9a188d3d0eea58d9d35dc866" gracePeriod=45 Nov 02 20:48:02.106625 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.106589 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:48:02.110365 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.110344 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:48:02.158660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.158628 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-tmp-dir\") pod \"4e44495a-efac-44a6-9077-724e39fa4d6f\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " Nov 02 20:48:02.159072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.158675 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-dshm\") pod \"4e44495a-efac-44a6-9077-724e39fa4d6f\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " Nov 02 20:48:02.159072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.158742 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e44495a-efac-44a6-9077-724e39fa4d6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e44495a-efac-44a6-9077-724e39fa4d6f-tls-certs\") pod \"4e44495a-efac-44a6-9077-724e39fa4d6f\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " Nov 02 20:48:02.159072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.158796 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e563127-e308-4d93-ba3a-46efea5469dc-kube-api-access-rrcnd\" (UniqueName: \"kubernetes.io/projected/1e563127-e308-4d93-ba3a-46efea5469dc-kube-api-access-rrcnd\") pod \"1e563127-e308-4d93-ba3a-46efea5469dc\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " Nov 02 20:48:02.159072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.158826 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-tmp-dir\") pod \"1e563127-e308-4d93-ba3a-46efea5469dc\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " Nov 02 20:48:02.159072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.158856 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-model-cache\") pod \"4e44495a-efac-44a6-9077-724e39fa4d6f\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " Nov 02 20:48:02.159072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.158893 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-model-cache\") pod \"1e563127-e308-4d93-ba3a-46efea5469dc\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " Nov 02 20:48:02.159072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.158927 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-dshm\") pod \"1e563127-e308-4d93-ba3a-46efea5469dc\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " Nov 02 20:48:02.159072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.158922 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-tmp-dir" pod "4e44495a-efac-44a6-9077-724e39fa4d6f" (UID: "4e44495a-efac-44a6-9077-724e39fa4d6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:02.159072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.159021 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e44495a-efac-44a6-9077-724e39fa4d6f-kube-api-access-8c5sh\" (UniqueName: \"kubernetes.io/projected/4e44495a-efac-44a6-9077-724e39fa4d6f-kube-api-access-8c5sh\") pod \"4e44495a-efac-44a6-9077-724e39fa4d6f\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " Nov 02 20:48:02.159595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.159077 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-home\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-home\") pod \"4e44495a-efac-44a6-9077-724e39fa4d6f\" (UID: \"4e44495a-efac-44a6-9077-724e39fa4d6f\") " Nov 02 20:48:02.159595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.159168 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-tmp-dir" pod "1e563127-e308-4d93-ba3a-46efea5469dc" (UID: "1e563127-e308-4d93-ba3a-46efea5469dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:02.159595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.159183 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-model-cache" pod "1e563127-e308-4d93-ba3a-46efea5469dc" (UID: "1e563127-e308-4d93-ba3a-46efea5469dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:02.159595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.159213 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e563127-e308-4d93-ba3a-46efea5469dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e563127-e308-4d93-ba3a-46efea5469dc-tls-certs\") pod \"1e563127-e308-4d93-ba3a-46efea5469dc\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " Nov 02 20:48:02.159595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.159245 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-home\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-home\") pod \"1e563127-e308-4d93-ba3a-46efea5469dc\" (UID: \"1e563127-e308-4d93-ba3a-46efea5469dc\") " Nov 02 20:48:02.159595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.159485 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-home" pod "4e44495a-efac-44a6-9077-724e39fa4d6f" (UID: "4e44495a-efac-44a6-9077-724e39fa4d6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:02.159896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.159700 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-home" pod "1e563127-e308-4d93-ba3a-46efea5469dc" (UID: "1e563127-e308-4d93-ba3a-46efea5469dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:02.161042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.160037 4971 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-home\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:02.161042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.160064 4971 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-tmp-dir\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:02.161042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.160099 4971 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-tmp-dir\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:02.161042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.160115 4971 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-model-cache\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:02.161042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.160129 4971 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-home\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:02.161042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.160754 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-model-cache" pod "4e44495a-efac-44a6-9077-724e39fa4d6f" (UID: "4e44495a-efac-44a6-9077-724e39fa4d6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:02.161978 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.161923 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e44495a-efac-44a6-9077-724e39fa4d6f-tls-certs" pod "4e44495a-efac-44a6-9077-724e39fa4d6f" (UID: "4e44495a-efac-44a6-9077-724e39fa4d6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:02.161978 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.161960 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e44495a-efac-44a6-9077-724e39fa4d6f-kube-api-access-8c5sh" pod "4e44495a-efac-44a6-9077-724e39fa4d6f" (UID: "4e44495a-efac-44a6-9077-724e39fa4d6f"). InnerVolumeSpecName "kube-api-access-8c5sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:02.162244 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.162067 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-dshm" pod "4e44495a-efac-44a6-9077-724e39fa4d6f" (UID: "4e44495a-efac-44a6-9077-724e39fa4d6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:02.162355 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.162338 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e563127-e308-4d93-ba3a-46efea5469dc-tls-certs" pod "1e563127-e308-4d93-ba3a-46efea5469dc" (UID: "1e563127-e308-4d93-ba3a-46efea5469dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:02.162712 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.162696 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e563127-e308-4d93-ba3a-46efea5469dc-kube-api-access-rrcnd" pod "1e563127-e308-4d93-ba3a-46efea5469dc" (UID: "1e563127-e308-4d93-ba3a-46efea5469dc"). InnerVolumeSpecName "kube-api-access-rrcnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:02.163024 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.163008 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-dshm" pod "1e563127-e308-4d93-ba3a-46efea5469dc" (UID: "1e563127-e308-4d93-ba3a-46efea5469dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:02.164676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.164655 4971 generic.go:356] "Generic (PLEG): container finished" podID="1e563127-e308-4d93-ba3a-46efea5469dc" containerID="69862c9a561a26a343e043387988201f1889fd6d9a188d3d0eea58d9d35dc866" exitCode=0 Nov 02 20:48:02.164777 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.164693 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" Nov 02 20:48:02.164777 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.164741 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" event={"ID":"1e563127-e308-4d93-ba3a-46efea5469dc","Type":"ContainerDied","Data":"69862c9a561a26a343e043387988201f1889fd6d9a188d3d0eea58d9d35dc866"} Nov 02 20:48:02.164851 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.164785 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb" event={"ID":"1e563127-e308-4d93-ba3a-46efea5469dc","Type":"ContainerDied","Data":"e0170eb7d182c9eb1ece194fa3a90a521ec3312f3b5a525f9c6f0d2f5c76304e"} Nov 02 20:48:02.164851 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.164804 4971 scope.go:122] "RemoveContainer" containerID="69862c9a561a26a343e043387988201f1889fd6d9a188d3d0eea58d9d35dc866" Nov 02 20:48:02.166383 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.166348 4971 generic.go:356] "Generic (PLEG): container finished" podID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerID="6c335aac4e8ab3d8c80453327d200e9734b824b441b0f9ab1eaef85540a20e30" exitCode=0 Nov 02 20:48:02.166512 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.166405 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" Nov 02 20:48:02.166512 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.166461 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" event={"ID":"4e44495a-efac-44a6-9077-724e39fa4d6f","Type":"ContainerDied","Data":"6c335aac4e8ab3d8c80453327d200e9734b824b441b0f9ab1eaef85540a20e30"} Nov 02 20:48:02.166631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.166511 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s" event={"ID":"4e44495a-efac-44a6-9077-724e39fa4d6f","Type":"ContainerDied","Data":"9408f144f927c31d0aa7c166f696647edc995cc84c1746f6a6dd0ef2feea0337"} Nov 02 20:48:02.176995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.176945 4971 scope.go:122] "RemoveContainer" containerID="69862c9a561a26a343e043387988201f1889fd6d9a188d3d0eea58d9d35dc866" Nov 02 20:48:02.177360 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:02.177233 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69862c9a561a26a343e043387988201f1889fd6d9a188d3d0eea58d9d35dc866\": container with ID starting with 69862c9a561a26a343e043387988201f1889fd6d9a188d3d0eea58d9d35dc866 not found: ID does not exist" containerID="69862c9a561a26a343e043387988201f1889fd6d9a188d3d0eea58d9d35dc866" Nov 02 20:48:02.177360 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.177269 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69862c9a561a26a343e043387988201f1889fd6d9a188d3d0eea58d9d35dc866"} err="failed to get container status \"69862c9a561a26a343e043387988201f1889fd6d9a188d3d0eea58d9d35dc866\": rpc error: code = NotFound desc = could not find container \"69862c9a561a26a343e043387988201f1889fd6d9a188d3d0eea58d9d35dc866\": container with ID starting with 69862c9a561a26a343e043387988201f1889fd6d9a188d3d0eea58d9d35dc866 not found: ID does not exist" Nov 02 20:48:02.177360 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.177285 4971 scope.go:122] "RemoveContainer" containerID="6c335aac4e8ab3d8c80453327d200e9734b824b441b0f9ab1eaef85540a20e30" Nov 02 20:48:02.192497 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.192466 4971 scope.go:122] "RemoveContainer" containerID="6c335aac4e8ab3d8c80453327d200e9734b824b441b0f9ab1eaef85540a20e30" Nov 02 20:48:02.193213 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:02.193189 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c335aac4e8ab3d8c80453327d200e9734b824b441b0f9ab1eaef85540a20e30\": container with ID starting with 6c335aac4e8ab3d8c80453327d200e9734b824b441b0f9ab1eaef85540a20e30 not found: ID does not exist" containerID="6c335aac4e8ab3d8c80453327d200e9734b824b441b0f9ab1eaef85540a20e30" Nov 02 20:48:02.193317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.193219 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c335aac4e8ab3d8c80453327d200e9734b824b441b0f9ab1eaef85540a20e30"} err="failed to get container status \"6c335aac4e8ab3d8c80453327d200e9734b824b441b0f9ab1eaef85540a20e30\": rpc error: code = NotFound desc = could not find container \"6c335aac4e8ab3d8c80453327d200e9734b824b441b0f9ab1eaef85540a20e30\": container with ID starting with 6c335aac4e8ab3d8c80453327d200e9734b824b441b0f9ab1eaef85540a20e30 not found: ID does not exist" Nov 02 20:48:02.261462 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.261424 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c5sh\" (UniqueName: \"kubernetes.io/projected/4e44495a-efac-44a6-9077-724e39fa4d6f-kube-api-access-8c5sh\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:02.261628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.261465 4971 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e563127-e308-4d93-ba3a-46efea5469dc-tls-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:02.261628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.261485 4971 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-dshm\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:02.261628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.261500 4971 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e44495a-efac-44a6-9077-724e39fa4d6f-tls-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:02.261628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.261515 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrcnd\" (UniqueName: \"kubernetes.io/projected/1e563127-e308-4d93-ba3a-46efea5469dc-kube-api-access-rrcnd\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:02.261628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.261529 4971 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e44495a-efac-44a6-9077-724e39fa4d6f-model-cache\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:02.261628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.261544 4971 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e563127-e308-4d93-ba3a-46efea5469dc-dshm\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:02.359213 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.359180 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" Nov 02 20:48:02.394301 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.394273 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s"] Nov 02 20:48:02.400314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.400273 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-30ee89/distinct-2-w0-30ee89-kserve-6d465c69b-sjg9s"] Nov 02 20:48:02.464007 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.463980 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9895c4c3-2e66-47a8-9d92-8172c289204b-kube-api-access-4phbb\" (UniqueName: \"kubernetes.io/projected/9895c4c3-2e66-47a8-9d92-8172c289204b-kube-api-access-4phbb\") pod \"9895c4c3-2e66-47a8-9d92-8172c289204b\" (UID: \"9895c4c3-2e66-47a8-9d92-8172c289204b\") " Nov 02 20:48:02.464187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.464028 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9895c4c3-2e66-47a8-9d92-8172c289204b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9895c4c3-2e66-47a8-9d92-8172c289204b-maas-api-xks-ca\") pod \"9895c4c3-2e66-47a8-9d92-8172c289204b\" (UID: \"9895c4c3-2e66-47a8-9d92-8172c289204b\") " Nov 02 20:48:02.464187 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.464166 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9895c4c3-2e66-47a8-9d92-8172c289204b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9895c4c3-2e66-47a8-9d92-8172c289204b-maas-api-openshift-ca\") pod \"9895c4c3-2e66-47a8-9d92-8172c289204b\" (UID: \"9895c4c3-2e66-47a8-9d92-8172c289204b\") " Nov 02 20:48:02.464488 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.464458 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9895c4c3-2e66-47a8-9d92-8172c289204b-maas-api-openshift-ca" pod "9895c4c3-2e66-47a8-9d92-8172c289204b" (UID: "9895c4c3-2e66-47a8-9d92-8172c289204b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:02.464660 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.464633 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9895c4c3-2e66-47a8-9d92-8172c289204b-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:02.466402 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.466371 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9895c4c3-2e66-47a8-9d92-8172c289204b-maas-api-xks-ca" pod "9895c4c3-2e66-47a8-9d92-8172c289204b" (UID: "9895c4c3-2e66-47a8-9d92-8172c289204b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:02.466477 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.466460 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9895c4c3-2e66-47a8-9d92-8172c289204b-kube-api-access-4phbb" pod "9895c4c3-2e66-47a8-9d92-8172c289204b" (UID: "9895c4c3-2e66-47a8-9d92-8172c289204b"). InnerVolumeSpecName "kube-api-access-4phbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:02.504661 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.504635 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb"] Nov 02 20:48:02.508022 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.507998 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-30ee89/premium-w0-30ee89-kserve-75d769457b-64xcb"] Nov 02 20:48:02.565484 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.565463 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4phbb\" (UniqueName: \"kubernetes.io/projected/9895c4c3-2e66-47a8-9d92-8172c289204b-kube-api-access-4phbb\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:02.565578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:02.565486 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9895c4c3-2e66-47a8-9d92-8172c289204b-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:03.160870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.160844 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:48:03.176116 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.176063 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" Nov 02 20:48:03.176282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.176095 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k" event={"ID":"9895c4c3-2e66-47a8-9d92-8172c289204b","Type":"ContainerDied","Data":"3829121501bf22e22f416f79b583784f7103d33690d1885af02591471b492742"} Nov 02 20:48:03.176282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.176232 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3829121501bf22e22f416f79b583784f7103d33690d1885af02591471b492742" Nov 02 20:48:03.178215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.178187 4971 generic.go:356] "Generic (PLEG): container finished" podID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" containerID="ae4a11cdd15d312e43afd1071dc944a972e76732dac3279b4a7569afea805341" exitCode=0 Nov 02 20:48:03.178371 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.178291 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" event={"ID":"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23","Type":"ContainerDied","Data":"ae4a11cdd15d312e43afd1071dc944a972e76732dac3279b4a7569afea805341"} Nov 02 20:48:03.178371 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.178299 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" Nov 02 20:48:03.178371 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.178326 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6" event={"ID":"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23","Type":"ContainerDied","Data":"b0e8aa58a291344eba4b434afbce91e5b10cca2bbebb96317d4229d31448c4ca"} Nov 02 20:48:03.178371 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.178347 4971 scope.go:122] "RemoveContainer" containerID="ae4a11cdd15d312e43afd1071dc944a972e76732dac3279b4a7569afea805341" Nov 02 20:48:03.192252 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.192234 4971 scope.go:122] "RemoveContainer" containerID="541c576ec9025f95e2e791888e95539786b459018e8def278a153f7ac0008cc5" Nov 02 20:48:03.201769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.201729 4971 scope.go:122] "RemoveContainer" containerID="ae4a11cdd15d312e43afd1071dc944a972e76732dac3279b4a7569afea805341" Nov 02 20:48:03.202067 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:03.202043 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae4a11cdd15d312e43afd1071dc944a972e76732dac3279b4a7569afea805341\": container with ID starting with ae4a11cdd15d312e43afd1071dc944a972e76732dac3279b4a7569afea805341 not found: ID does not exist" containerID="ae4a11cdd15d312e43afd1071dc944a972e76732dac3279b4a7569afea805341" Nov 02 20:48:03.202168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.202074 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae4a11cdd15d312e43afd1071dc944a972e76732dac3279b4a7569afea805341"} err="failed to get container status \"ae4a11cdd15d312e43afd1071dc944a972e76732dac3279b4a7569afea805341\": rpc error: code = NotFound desc = could not find container \"ae4a11cdd15d312e43afd1071dc944a972e76732dac3279b4a7569afea805341\": container with ID starting with ae4a11cdd15d312e43afd1071dc944a972e76732dac3279b4a7569afea805341 not found: ID does not exist" Nov 02 20:48:03.202168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.202119 4971 scope.go:122] "RemoveContainer" containerID="541c576ec9025f95e2e791888e95539786b459018e8def278a153f7ac0008cc5" Nov 02 20:48:03.202414 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:03.202392 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541c576ec9025f95e2e791888e95539786b459018e8def278a153f7ac0008cc5\": container with ID starting with 541c576ec9025f95e2e791888e95539786b459018e8def278a153f7ac0008cc5 not found: ID does not exist" containerID="541c576ec9025f95e2e791888e95539786b459018e8def278a153f7ac0008cc5" Nov 02 20:48:03.202470 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.202421 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541c576ec9025f95e2e791888e95539786b459018e8def278a153f7ac0008cc5"} err="failed to get container status \"541c576ec9025f95e2e791888e95539786b459018e8def278a153f7ac0008cc5\": rpc error: code = NotFound desc = could not find container \"541c576ec9025f95e2e791888e95539786b459018e8def278a153f7ac0008cc5\": container with ID starting with 541c576ec9025f95e2e791888e95539786b459018e8def278a153f7ac0008cc5 not found: ID does not exist" Nov 02 20:48:03.274552 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.274523 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-podinfo\") pod \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " Nov 02 20:48:03.274707 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.274607 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-data\") pod \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " Nov 02 20:48:03.274707 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.274644 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-workload-certs\") pod \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " Nov 02 20:48:03.274828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.274702 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-kube-api-access-bsc5v\" (UniqueName: \"kubernetes.io/projected/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-kube-api-access-bsc5v\") pod \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " Nov 02 20:48:03.274828 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.274795 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istiod-ca-cert\") pod \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " Nov 02 20:48:03.274938 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.274826 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-workload-socket\") pod \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " Nov 02 20:48:03.274938 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.274899 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-credential-socket\") pod \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " Nov 02 20:48:03.275041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.274950 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-envoy\") pod \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " Nov 02 20:48:03.275041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.274950 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-data" pod "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" (UID: "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:03.275041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.274965 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-workload-certs" pod "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" (UID: "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:03.275041 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.275015 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-token\" (UniqueName: \"kubernetes.io/projected/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-token\") pod \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\" (UID: \"ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23\") " Nov 02 20:48:03.275364 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.275335 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istiod-ca-cert" pod "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" (UID: "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:03.275601 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.275416 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-credential-socket" pod "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" (UID: "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:03.275904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.275879 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-workload-socket" pod "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" (UID: "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:03.275904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.275897 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:03.276128 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.275923 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:03.276128 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.275938 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:03.276128 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.275953 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:03.277799 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.277775 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-podinfo" pod "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" (UID: "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:48:03.277917 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.277816 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-kube-api-access-bsc5v" pod "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" (UID: "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23"). InnerVolumeSpecName "kube-api-access-bsc5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:03.278048 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.278027 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-token" pod "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" (UID: "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:03.278153 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.278136 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-envoy" pod "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" (UID: "ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:03.293139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.293115 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n"] Nov 02 20:48:03.346594 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:03.346558 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:48:03.376966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.376932 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:03.376966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.376969 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:03.377278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.376985 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:03.377278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.376999 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:03.377278 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.377013 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsc5v\" (UniqueName: \"kubernetes.io/projected/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23-kube-api-access-bsc5v\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:03.498960 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.498929 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:03.651280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.651193 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6"] Nov 02 20:48:03.656928 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.656906 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-039b89-openshift-default-6fffcf4dd5-ppvf6"] Nov 02 20:48:03.866919 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.866887 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" path="/var/lib/kubelet/pods/1e563127-e308-4d93-ba3a-46efea5469dc/volumes" Nov 02 20:48:03.867419 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.867404 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" path="/var/lib/kubelet/pods/4e44495a-efac-44a6-9077-724e39fa4d6f/volumes" Nov 02 20:48:03.867871 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:03.867858 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" path="/var/lib/kubelet/pods/ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23/volumes" Nov 02 20:48:06.029421 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:06.029338 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:07.602576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:07.602540 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" podUID="5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:48:07.649585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:07.649556 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:07.908279 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:07.908187 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:08.049196 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:08.049160 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t"] Nov 02 20:48:08.095838 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:08.095806 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-1620dc-sbk9t"] Nov 02 20:48:08.157769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:08.157736 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:08.553909 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:08.553878 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:09.579578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:09.579540 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:09.827670 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:09.827625 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:09.866346 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:09.866311 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5305b6-a8a8-4a7a-854b-233591f338d0" path="/var/lib/kubelet/pods/6a5305b6-a8a8-4a7a-854b-233591f338d0/volumes" Nov 02 20:48:10.384758 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:10.378639 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t"] Nov 02 20:48:10.492442 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:10.492295 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558"] Nov 02 20:48:10.863655 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:10.863624 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:11.115861 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:11.115782 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:13.116753 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:13.116711 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:14.443407 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:14.443372 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:14.703371 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:14.703288 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:14.720232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:14.720203 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr"] Nov 02 20:48:14.720467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:14.720440 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" podUID="fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" containerName="istio-proxy" containerID="cri-o://e8241a558f111b3357ad5a7ef276dc4ac2884ab7b1669596ce083130d7640c0c" gracePeriod=30 Nov 02 20:48:14.953584 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:14.953497 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:15.379626 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.379574 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" podUID="a29a43c7-6192-4151-9c21-bcedab45ba8c" containerName="payload-processing" containerID="cri-o://47065991283d41332d9dd31f44e4252ec6be2c9b4c15f51df5d8f9d2ef3da751" gracePeriod=25 Nov 02 20:48:15.493499 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.493463 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" podUID="ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48" containerName="payload-pre-processing" containerID="cri-o://da848c22a288dc7a52c88904ff7c4c9dc75a9b6b7556a00424b59c76eaa85466" gracePeriod=25 Nov 02 20:48:15.548325 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.548297 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc"] Nov 02 20:48:15.549014 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.548994 4971 cpu_manager.go:414] "containerMap: removing container" podUID="9895c4c3-2e66-47a8-9d92-8172c289204b" containerName="revoke-keys" Nov 02 20:48:15.549148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.549020 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9895c4c3-2e66-47a8-9d92-8172c289204b" containerName="revoke-keys" Nov 02 20:48:15.549148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.549041 4971 cpu_manager.go:414] "containerMap: removing container" podUID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" containerName="istio-proxy" Nov 02 20:48:15.549148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.549046 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" containerName="istio-proxy" Nov 02 20:48:15.549148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.549056 4971 cpu_manager.go:414] "containerMap: removing container" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" Nov 02 20:48:15.549148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.549062 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" Nov 02 20:48:15.549148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.549074 4971 cpu_manager.go:414] "containerMap: removing container" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" Nov 02 20:48:15.549148 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.549094 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" Nov 02 20:48:15.549405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.549204 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e563127-e308-4d93-ba3a-46efea5469dc" containerName="main" Nov 02 20:48:15.549405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.549216 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e44495a-efac-44a6-9077-724e39fa4d6f" containerName="main" Nov 02 20:48:15.549405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.549224 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" containerName="istio-proxy" Nov 02 20:48:15.549405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.549232 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="9895c4c3-2e66-47a8-9d92-8172c289204b" containerName="revoke-keys" Nov 02 20:48:15.549405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.549240 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" containerName="istio-proxy" Nov 02 20:48:15.554339 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.554321 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" Nov 02 20:48:15.609890 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.609854 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc"] Nov 02 20:48:15.679076 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.679017 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:48:15.679076 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.679019 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:48:15.735859 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.735835 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985b395c-deed-4e79-b13e-5d8566f29218-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc\" (UID: \"985b395c-deed-4e79-b13e-5d8566f29218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" Nov 02 20:48:15.735980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.735958 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985b395c-deed-4e79-b13e-5d8566f29218-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc\" (UID: \"985b395c-deed-4e79-b13e-5d8566f29218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" Nov 02 20:48:15.736036 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.736021 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx6bh\" (UniqueName: \"kubernetes.io/projected/985b395c-deed-4e79-b13e-5d8566f29218-kube-api-access-gx6bh\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc\" (UID: \"985b395c-deed-4e79-b13e-5d8566f29218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" Nov 02 20:48:15.837521 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.837491 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985b395c-deed-4e79-b13e-5d8566f29218-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc\" (UID: \"985b395c-deed-4e79-b13e-5d8566f29218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" Nov 02 20:48:15.837648 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.837580 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985b395c-deed-4e79-b13e-5d8566f29218-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc\" (UID: \"985b395c-deed-4e79-b13e-5d8566f29218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" Nov 02 20:48:15.837648 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.837603 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx6bh\" (UniqueName: \"kubernetes.io/projected/985b395c-deed-4e79-b13e-5d8566f29218-kube-api-access-gx6bh\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc\" (UID: \"985b395c-deed-4e79-b13e-5d8566f29218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" Nov 02 20:48:15.838229 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.838205 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985b395c-deed-4e79-b13e-5d8566f29218-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc\" (UID: \"985b395c-deed-4e79-b13e-5d8566f29218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" Nov 02 20:48:15.839998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.839976 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985b395c-deed-4e79-b13e-5d8566f29218-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc\" (UID: \"985b395c-deed-4e79-b13e-5d8566f29218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" Nov 02 20:48:15.895347 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:15.895324 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx6bh\" (UniqueName: \"kubernetes.io/projected/985b395c-deed-4e79-b13e-5d8566f29218-kube-api-access-gx6bh\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc\" (UID: \"985b395c-deed-4e79-b13e-5d8566f29218\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" Nov 02 20:48:16.168597 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:16.168571 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" Nov 02 20:48:16.537789 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:16.537756 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc"] Nov 02 20:48:16.538267 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:48:16.538242 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985b395c_deed_4e79_b13e_5d8566f29218.slice/crio-73a87f67b292e0273b7ef5ab5d3f20bacef17885c1add9a9e9b7785f16adfca7 WatchSource:0}: Error finding container 73a87f67b292e0273b7ef5ab5d3f20bacef17885c1add9a9e9b7785f16adfca7: Status 404 returned error can't find the container with id 73a87f67b292e0273b7ef5ab5d3f20bacef17885c1add9a9e9b7785f16adfca7 Nov 02 20:48:17.299953 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:17.299913 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" event={"ID":"985b395c-deed-4e79-b13e-5d8566f29218","Type":"ContainerStarted","Data":"b47fdc476e7ab895dd5a3e8bc51818234a73ca9bbbd31f8de711f917126a98ba"} Nov 02 20:48:17.300162 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:17.299982 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" event={"ID":"985b395c-deed-4e79-b13e-5d8566f29218","Type":"ContainerStarted","Data":"73a87f67b292e0273b7ef5ab5d3f20bacef17885c1add9a9e9b7785f16adfca7"} Nov 02 20:48:17.412094 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:17.412026 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" podStartSLOduration=2.412012227 podStartE2EDuration="2.412012227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:48:17.408872902 +0000 UTC m=+2025.698529430" watchObservedRunningTime="2026-11-02 20:48:17.412012227 +0000 UTC m=+2025.701668754" Nov 02 20:48:17.602026 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:17.601993 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" podUID="5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:48:18.138994 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.138961 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:18.166258 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.166232 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" Nov 02 20:48:18.263799 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.263744 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4-kube-api-access-5p5ng\" (UniqueName: \"kubernetes.io/projected/5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4-kube-api-access-5p5ng\") pod \"5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4\" (UID: \"5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4\") " Nov 02 20:48:18.263799 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.263783 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4-plugins-config-volume\") pod \"5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4\" (UID: \"5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4\") " Nov 02 20:48:18.264208 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.264182 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4-plugins-config-volume" pod "5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4" (UID: "5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:18.266017 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.265994 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4-kube-api-access-5p5ng" pod "5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4" (UID: "5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4"). InnerVolumeSpecName "kube-api-access-5p5ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:18.309712 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.309675 4971 generic.go:356] "Generic (PLEG): container finished" podID="985b395c-deed-4e79-b13e-5d8566f29218" containerID="b47fdc476e7ab895dd5a3e8bc51818234a73ca9bbbd31f8de711f917126a98ba" exitCode=0 Nov 02 20:48:18.309876 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.309756 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" event={"ID":"985b395c-deed-4e79-b13e-5d8566f29218","Type":"ContainerDied","Data":"b47fdc476e7ab895dd5a3e8bc51818234a73ca9bbbd31f8de711f917126a98ba"} Nov 02 20:48:18.311546 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.311522 4971 generic.go:356] "Generic (PLEG): container finished" podID="5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4" containerID="3ff6e63ca94b047be44ad0a31a8f5c4c01d23d64769a882a6c104efdb78c184e" exitCode=0 Nov 02 20:48:18.311671 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.311566 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" Nov 02 20:48:18.311671 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.311608 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" event={"ID":"5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4","Type":"ContainerDied","Data":"3ff6e63ca94b047be44ad0a31a8f5c4c01d23d64769a882a6c104efdb78c184e"} Nov 02 20:48:18.311671 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.311662 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs" event={"ID":"5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4","Type":"ContainerDied","Data":"86bee941de3dfe309658c4104bfe0a74cb8f934c2477f9c8b8a90c214af10046"} Nov 02 20:48:18.311827 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.311694 4971 scope.go:122] "RemoveContainer" containerID="3ff6e63ca94b047be44ad0a31a8f5c4c01d23d64769a882a6c104efdb78c184e" Nov 02 20:48:18.322192 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.321560 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" containerID="cri-o://11d3d4c4d4779e4870a068882da070abaa134ad00515be3b05ec631b3dffab9a" gracePeriod=45 Nov 02 20:48:18.324839 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.324822 4971 scope.go:122] "RemoveContainer" containerID="3ff6e63ca94b047be44ad0a31a8f5c4c01d23d64769a882a6c104efdb78c184e" Nov 02 20:48:18.325272 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:18.325242 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff6e63ca94b047be44ad0a31a8f5c4c01d23d64769a882a6c104efdb78c184e\": container with ID starting with 3ff6e63ca94b047be44ad0a31a8f5c4c01d23d64769a882a6c104efdb78c184e not found: ID does not exist" containerID="3ff6e63ca94b047be44ad0a31a8f5c4c01d23d64769a882a6c104efdb78c184e" Nov 02 20:48:18.325348 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.325282 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff6e63ca94b047be44ad0a31a8f5c4c01d23d64769a882a6c104efdb78c184e"} err="failed to get container status \"3ff6e63ca94b047be44ad0a31a8f5c4c01d23d64769a882a6c104efdb78c184e\": rpc error: code = NotFound desc = could not find container \"3ff6e63ca94b047be44ad0a31a8f5c4c01d23d64769a882a6c104efdb78c184e\": container with ID starting with 3ff6e63ca94b047be44ad0a31a8f5c4c01d23d64769a882a6c104efdb78c184e not found: ID does not exist" Nov 02 20:48:18.364803 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.364782 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:18.364803 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.364804 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p5ng\" (UniqueName: \"kubernetes.io/projected/5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4-kube-api-access-5p5ng\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:18.409349 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.409324 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": dial tcp 10.132.0.115:8000: connect: connection refused" Nov 02 20:48:18.528046 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.527975 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs"] Nov 02 20:48:18.535068 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.535041 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-30ee89-69b4488f67-bkgbs"] Nov 02 20:48:18.585377 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.585354 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:48:18.667211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.667183 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-model-cache\") pod \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " Nov 02 20:48:18.667559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.667224 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-home\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-home\") pod \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " Nov 02 20:48:18.667559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.667286 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-dshm\") pod \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " Nov 02 20:48:18.667559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.667384 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-kube-api-access-fhgkg\" (UniqueName: \"kubernetes.io/projected/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-kube-api-access-fhgkg\") pod \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " Nov 02 20:48:18.667559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.667453 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-model-cache" pod "b7fe7734-b7d6-497e-818a-aa48fe34d4c9" (UID: "b7fe7734-b7d6-497e-818a-aa48fe34d4c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:18.667559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.667458 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-tls-certs\") pod \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " Nov 02 20:48:18.667559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.667487 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-home" pod "b7fe7734-b7d6-497e-818a-aa48fe34d4c9" (UID: "b7fe7734-b7d6-497e-818a-aa48fe34d4c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:18.667559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.667521 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-tmp-dir\") pod \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\" (UID: \"b7fe7734-b7d6-497e-818a-aa48fe34d4c9\") " Nov 02 20:48:18.667866 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.667722 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-tmp-dir" pod "b7fe7734-b7d6-497e-818a-aa48fe34d4c9" (UID: "b7fe7734-b7d6-497e-818a-aa48fe34d4c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:18.668016 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.667997 4971 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-tmp-dir\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:18.668304 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.668022 4971 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-model-cache\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:18.668304 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.668063 4971 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-home\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:18.669794 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.669773 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-dshm" pod "b7fe7734-b7d6-497e-818a-aa48fe34d4c9" (UID: "b7fe7734-b7d6-497e-818a-aa48fe34d4c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:18.669896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.669825 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-tls-certs" pod "b7fe7734-b7d6-497e-818a-aa48fe34d4c9" (UID: "b7fe7734-b7d6-497e-818a-aa48fe34d4c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:18.669896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.669837 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-kube-api-access-fhgkg" pod "b7fe7734-b7d6-497e-818a-aa48fe34d4c9" (UID: "b7fe7734-b7d6-497e-818a-aa48fe34d4c9"). InnerVolumeSpecName "kube-api-access-fhgkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:18.768934 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.768901 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhgkg\" (UniqueName: \"kubernetes.io/projected/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-kube-api-access-fhgkg\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:18.769130 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.768936 4971 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-tls-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:18.769130 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:18.768954 4971 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7fe7734-b7d6-497e-818a-aa48fe34d4c9-dshm\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:19.322251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.322176 4971 generic.go:356] "Generic (PLEG): container finished" podID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerID="11d3d4c4d4779e4870a068882da070abaa134ad00515be3b05ec631b3dffab9a" exitCode=0 Nov 02 20:48:19.322251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.322250 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" Nov 02 20:48:19.322507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.322262 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" event={"ID":"b7fe7734-b7d6-497e-818a-aa48fe34d4c9","Type":"ContainerDied","Data":"11d3d4c4d4779e4870a068882da070abaa134ad00515be3b05ec631b3dffab9a"} Nov 02 20:48:19.322507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.322308 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n" event={"ID":"b7fe7734-b7d6-497e-818a-aa48fe34d4c9","Type":"ContainerDied","Data":"3193c35d4b8e8c7ff99daaf0ebe525c4df4a73bacf8f485f2346a3dac6ad8f1a"} Nov 02 20:48:19.322507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.322325 4971 scope.go:122] "RemoveContainer" containerID="11d3d4c4d4779e4870a068882da070abaa134ad00515be3b05ec631b3dffab9a" Nov 02 20:48:19.334293 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.334272 4971 scope.go:122] "RemoveContainer" containerID="11d3d4c4d4779e4870a068882da070abaa134ad00515be3b05ec631b3dffab9a" Nov 02 20:48:19.334576 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:19.334556 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d3d4c4d4779e4870a068882da070abaa134ad00515be3b05ec631b3dffab9a\": container with ID starting with 11d3d4c4d4779e4870a068882da070abaa134ad00515be3b05ec631b3dffab9a not found: ID does not exist" containerID="11d3d4c4d4779e4870a068882da070abaa134ad00515be3b05ec631b3dffab9a" Nov 02 20:48:19.334652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.334587 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d3d4c4d4779e4870a068882da070abaa134ad00515be3b05ec631b3dffab9a"} err="failed to get container status \"11d3d4c4d4779e4870a068882da070abaa134ad00515be3b05ec631b3dffab9a\": rpc error: code = NotFound desc = could not find container \"11d3d4c4d4779e4870a068882da070abaa134ad00515be3b05ec631b3dffab9a\": container with ID starting with 11d3d4c4d4779e4870a068882da070abaa134ad00515be3b05ec631b3dffab9a not found: ID does not exist" Nov 02 20:48:19.472844 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.472820 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" Nov 02 20:48:19.478358 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.478336 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n"] Nov 02 20:48:19.484517 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.484492 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-2250b4/distinct-2-w1-2250b4-kserve-b7cf64cb4-pv74n"] Nov 02 20:48:19.576904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.576851 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/985b395c-deed-4e79-b13e-5d8566f29218-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985b395c-deed-4e79-b13e-5d8566f29218-maas-api-openshift-ca\") pod \"985b395c-deed-4e79-b13e-5d8566f29218\" (UID: \"985b395c-deed-4e79-b13e-5d8566f29218\") " Nov 02 20:48:19.577016 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.576941 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/985b395c-deed-4e79-b13e-5d8566f29218-kube-api-access-gx6bh\" (UniqueName: \"kubernetes.io/projected/985b395c-deed-4e79-b13e-5d8566f29218-kube-api-access-gx6bh\") pod \"985b395c-deed-4e79-b13e-5d8566f29218\" (UID: \"985b395c-deed-4e79-b13e-5d8566f29218\") " Nov 02 20:48:19.577064 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.577022 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/985b395c-deed-4e79-b13e-5d8566f29218-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985b395c-deed-4e79-b13e-5d8566f29218-maas-api-xks-ca\") pod \"985b395c-deed-4e79-b13e-5d8566f29218\" (UID: \"985b395c-deed-4e79-b13e-5d8566f29218\") " Nov 02 20:48:19.577252 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.577228 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985b395c-deed-4e79-b13e-5d8566f29218-maas-api-openshift-ca" pod "985b395c-deed-4e79-b13e-5d8566f29218" (UID: "985b395c-deed-4e79-b13e-5d8566f29218"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:19.577461 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.577438 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985b395c-deed-4e79-b13e-5d8566f29218-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:19.579282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.579254 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/985b395c-deed-4e79-b13e-5d8566f29218-maas-api-xks-ca" pod "985b395c-deed-4e79-b13e-5d8566f29218" (UID: "985b395c-deed-4e79-b13e-5d8566f29218"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:19.579282 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.579267 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985b395c-deed-4e79-b13e-5d8566f29218-kube-api-access-gx6bh" pod "985b395c-deed-4e79-b13e-5d8566f29218" (UID: "985b395c-deed-4e79-b13e-5d8566f29218"). InnerVolumeSpecName "kube-api-access-gx6bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:19.678762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.678736 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx6bh\" (UniqueName: \"kubernetes.io/projected/985b395c-deed-4e79-b13e-5d8566f29218-kube-api-access-gx6bh\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:19.678762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.678763 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985b395c-deed-4e79-b13e-5d8566f29218-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:19.867581 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.867548 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4" path="/var/lib/kubelet/pods/5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4/volumes" Nov 02 20:48:19.868056 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.868042 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" path="/var/lib/kubelet/pods/b7fe7734-b7d6-497e-818a-aa48fe34d4c9/volumes" Nov 02 20:48:19.978042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:19.978016 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:48:20.082327 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.082290 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-envoy\") pod \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " Nov 02 20:48:20.082479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.082344 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-podinfo\") pod \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " Nov 02 20:48:20.082479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.082368 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-data\") pod \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " Nov 02 20:48:20.082479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.082426 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-workload-certs\") pod \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " Nov 02 20:48:20.082479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.082476 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-kube-api-access-xkkrn\" (UniqueName: \"kubernetes.io/projected/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-kube-api-access-xkkrn\") pod \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " Nov 02 20:48:20.082722 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.082524 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istiod-ca-cert\") pod \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " Nov 02 20:48:20.082722 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.082568 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-credential-socket\") pod \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " Nov 02 20:48:20.082722 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.082638 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-token\" (UniqueName: \"kubernetes.io/projected/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-token\") pod \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " Nov 02 20:48:20.082722 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.082671 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-workload-socket\") pod \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\" (UID: \"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284\") " Nov 02 20:48:20.082945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.082707 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-data" pod "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" (UID: "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:20.082945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.082780 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-workload-certs" pod "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" (UID: "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:20.083060 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.083024 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-workload-socket" pod "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" (UID: "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:20.083317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.083046 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-credential-socket" pod "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" (UID: "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:20.083317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.083159 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:20.083317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.083177 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:20.083317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.083185 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:20.083317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.083193 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:20.083623 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.083390 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istiod-ca-cert" pod "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" (UID: "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:20.084713 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.084684 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-podinfo" pod "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" (UID: "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:48:20.084853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.084832 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-envoy" pod "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" (UID: "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:20.084920 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.084903 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-kube-api-access-xkkrn" pod "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" (UID: "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284"). InnerVolumeSpecName "kube-api-access-xkkrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:20.085235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.085215 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-token" pod "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" (UID: "fe1f0129-11cb-4e2b-a6bc-d04f12b3b284"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:20.184767 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.184731 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkkrn\" (UniqueName: \"kubernetes.io/projected/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-kube-api-access-xkkrn\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:20.184767 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.184762 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:20.184948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.184775 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:20.184948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.184787 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:20.184948 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.184800 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:20.333497 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.333466 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" Nov 02 20:48:20.333659 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.333493 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc" event={"ID":"985b395c-deed-4e79-b13e-5d8566f29218","Type":"ContainerDied","Data":"73a87f67b292e0273b7ef5ab5d3f20bacef17885c1add9a9e9b7785f16adfca7"} Nov 02 20:48:20.333659 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.333546 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73a87f67b292e0273b7ef5ab5d3f20bacef17885c1add9a9e9b7785f16adfca7" Nov 02 20:48:20.335440 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.335418 4971 generic.go:356] "Generic (PLEG): container finished" podID="fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" containerID="e8241a558f111b3357ad5a7ef276dc4ac2884ab7b1669596ce083130d7640c0c" exitCode=0 Nov 02 20:48:20.335553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.335478 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" Nov 02 20:48:20.335553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.335501 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" event={"ID":"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284","Type":"ContainerDied","Data":"e8241a558f111b3357ad5a7ef276dc4ac2884ab7b1669596ce083130d7640c0c"} Nov 02 20:48:20.335553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.335547 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr" event={"ID":"fe1f0129-11cb-4e2b-a6bc-d04f12b3b284","Type":"ContainerDied","Data":"7104e5cc96c1bd0022bab3a99b210c2f5577d2e66482d1f31e2fa2f6769492e5"} Nov 02 20:48:20.335727 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.335578 4971 scope.go:122] "RemoveContainer" containerID="e8241a558f111b3357ad5a7ef276dc4ac2884ab7b1669596ce083130d7640c0c" Nov 02 20:48:20.347875 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.347859 4971 scope.go:122] "RemoveContainer" containerID="5cd41de231662603c3d546f5a38b51de3af269c7b4d46c17c254ac92116ddf07" Nov 02 20:48:20.357308 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.357291 4971 scope.go:122] "RemoveContainer" containerID="e8241a558f111b3357ad5a7ef276dc4ac2884ab7b1669596ce083130d7640c0c" Nov 02 20:48:20.357569 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:20.357549 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8241a558f111b3357ad5a7ef276dc4ac2884ab7b1669596ce083130d7640c0c\": container with ID starting with e8241a558f111b3357ad5a7ef276dc4ac2884ab7b1669596ce083130d7640c0c not found: ID does not exist" containerID="e8241a558f111b3357ad5a7ef276dc4ac2884ab7b1669596ce083130d7640c0c" Nov 02 20:48:20.357646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.357581 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8241a558f111b3357ad5a7ef276dc4ac2884ab7b1669596ce083130d7640c0c"} err="failed to get container status \"e8241a558f111b3357ad5a7ef276dc4ac2884ab7b1669596ce083130d7640c0c\": rpc error: code = NotFound desc = could not find container \"e8241a558f111b3357ad5a7ef276dc4ac2884ab7b1669596ce083130d7640c0c\": container with ID starting with e8241a558f111b3357ad5a7ef276dc4ac2884ab7b1669596ce083130d7640c0c not found: ID does not exist" Nov 02 20:48:20.357646 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.357604 4971 scope.go:122] "RemoveContainer" containerID="5cd41de231662603c3d546f5a38b51de3af269c7b4d46c17c254ac92116ddf07" Nov 02 20:48:20.357857 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:20.357839 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd41de231662603c3d546f5a38b51de3af269c7b4d46c17c254ac92116ddf07\": container with ID starting with 5cd41de231662603c3d546f5a38b51de3af269c7b4d46c17c254ac92116ddf07 not found: ID does not exist" containerID="5cd41de231662603c3d546f5a38b51de3af269c7b4d46c17c254ac92116ddf07" Nov 02 20:48:20.357895 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.357864 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd41de231662603c3d546f5a38b51de3af269c7b4d46c17c254ac92116ddf07"} err="failed to get container status \"5cd41de231662603c3d546f5a38b51de3af269c7b4d46c17c254ac92116ddf07\": rpc error: code = NotFound desc = could not find container \"5cd41de231662603c3d546f5a38b51de3af269c7b4d46c17c254ac92116ddf07\": container with ID starting with 5cd41de231662603c3d546f5a38b51de3af269c7b4d46c17c254ac92116ddf07 not found: ID does not exist" Nov 02 20:48:20.542317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.542287 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr"] Nov 02 20:48:20.546481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:20.546459 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-1620dc-openshift-default-7964cdcf79-vwssr"] Nov 02 20:48:21.786901 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:21.786865 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" podUID="ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:48:21.787348 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:21.786916 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" podUID="a29a43c7-6192-4151-9c21-bcedab45ba8c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:48:21.866578 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:21.866549 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" path="/var/lib/kubelet/pods/fe1f0129-11cb-4e2b-a6bc-d04f12b3b284/volumes" Nov 02 20:48:21.960475 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:21.960443 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:22.209313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.209279 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:22.386467 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.386437 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t"] Nov 02 20:48:22.387233 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387210 4971 cpu_manager.go:414] "containerMap: removing container" podUID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" containerName="istio-proxy" Nov 02 20:48:22.387387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387235 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1e4721-0a73-49fe-83c5-2ad7a7ba0a23" containerName="istio-proxy" Nov 02 20:48:22.387387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387252 4971 cpu_manager.go:414] "containerMap: removing container" podUID="5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4" containerName="payload-pre-processing" Nov 02 20:48:22.387387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387258 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4" containerName="payload-pre-processing" Nov 02 20:48:22.387387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387277 4971 cpu_manager.go:414] "containerMap: removing container" podUID="fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" containerName="istio-proxy" Nov 02 20:48:22.387387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387283 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" containerName="istio-proxy" Nov 02 20:48:22.387387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387296 4971 cpu_manager.go:414] "containerMap: removing container" podUID="985b395c-deed-4e79-b13e-5d8566f29218" containerName="revoke-keys" Nov 02 20:48:22.387387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387301 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="985b395c-deed-4e79-b13e-5d8566f29218" containerName="revoke-keys" Nov 02 20:48:22.387387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387309 4971 cpu_manager.go:414] "containerMap: removing container" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" Nov 02 20:48:22.387387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387313 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" Nov 02 20:48:22.387387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387323 4971 cpu_manager.go:414] "containerMap: removing container" podUID="fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" containerName="istio-proxy" Nov 02 20:48:22.387387 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387327 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" containerName="istio-proxy" Nov 02 20:48:22.387775 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387419 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ccd73f1-7600-4593-8ee4-f6c00d6ea0f4" containerName="payload-pre-processing" Nov 02 20:48:22.387775 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387431 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" containerName="istio-proxy" Nov 02 20:48:22.387775 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387438 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="985b395c-deed-4e79-b13e-5d8566f29218" containerName="revoke-keys" Nov 02 20:48:22.387775 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387446 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7fe7734-b7d6-497e-818a-aa48fe34d4c9" containerName="main" Nov 02 20:48:22.387775 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.387452 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1f0129-11cb-4e2b-a6bc-d04f12b3b284" containerName="istio-proxy" Nov 02 20:48:22.394874 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.394854 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" Nov 02 20:48:22.442823 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.442785 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6"] Nov 02 20:48:22.447006 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.446989 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6" Nov 02 20:48:22.486073 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.486004 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:48:22.486214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.486116 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:48:22.489371 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.489353 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t"] Nov 02 20:48:22.503434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.503415 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6"] Nov 02 20:48:22.513123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.512236 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be58e326-8b22-4a60-a9fb-0a4546a54ae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e3fffe3-vdv8t\" (UID: \"be58e326-8b22-4a60-a9fb-0a4546a54ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" Nov 02 20:48:22.513123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.512283 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be58e326-8b22-4a60-a9fb-0a4546a54ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e3fffe3-vdv8t\" (UID: \"be58e326-8b22-4a60-a9fb-0a4546a54ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" Nov 02 20:48:22.513123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.512380 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fzcb\" (UniqueName: \"kubernetes.io/projected/be58e326-8b22-4a60-a9fb-0a4546a54ae7-kube-api-access-2fzcb\") pod \"maas-api-revoke-sub-6e3fffe3-vdv8t\" (UID: \"be58e326-8b22-4a60-a9fb-0a4546a54ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" Nov 02 20:48:22.513123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.512439 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg4hs\" (UniqueName: \"kubernetes.io/projected/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-kube-api-access-qg4hs\") pod \"maas-api-revoke-sub-fef94b89-94kk6\" (UID: \"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6" Nov 02 20:48:22.513123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.512551 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef94b89-94kk6\" (UID: \"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6" Nov 02 20:48:22.513123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.512591 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef94b89-94kk6\" (UID: \"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6" Nov 02 20:48:22.613958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.613928 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef94b89-94kk6\" (UID: \"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6" Nov 02 20:48:22.614121 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.613972 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef94b89-94kk6\" (UID: \"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6" Nov 02 20:48:22.614223 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.614150 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be58e326-8b22-4a60-a9fb-0a4546a54ae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e3fffe3-vdv8t\" (UID: \"be58e326-8b22-4a60-a9fb-0a4546a54ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" Nov 02 20:48:22.614223 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.614194 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be58e326-8b22-4a60-a9fb-0a4546a54ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e3fffe3-vdv8t\" (UID: \"be58e326-8b22-4a60-a9fb-0a4546a54ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" Nov 02 20:48:22.614343 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.614234 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fzcb\" (UniqueName: \"kubernetes.io/projected/be58e326-8b22-4a60-a9fb-0a4546a54ae7-kube-api-access-2fzcb\") pod \"maas-api-revoke-sub-6e3fffe3-vdv8t\" (UID: \"be58e326-8b22-4a60-a9fb-0a4546a54ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" Nov 02 20:48:22.614343 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.614277 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg4hs\" (UniqueName: \"kubernetes.io/projected/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-kube-api-access-qg4hs\") pod \"maas-api-revoke-sub-fef94b89-94kk6\" (UID: \"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6" Nov 02 20:48:22.614715 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.614690 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef94b89-94kk6\" (UID: \"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6" Nov 02 20:48:22.614946 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.614889 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be58e326-8b22-4a60-a9fb-0a4546a54ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e3fffe3-vdv8t\" (UID: \"be58e326-8b22-4a60-a9fb-0a4546a54ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" Nov 02 20:48:22.616637 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.616607 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef94b89-94kk6\" (UID: \"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6" Nov 02 20:48:22.616725 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.616618 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be58e326-8b22-4a60-a9fb-0a4546a54ae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e3fffe3-vdv8t\" (UID: \"be58e326-8b22-4a60-a9fb-0a4546a54ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" Nov 02 20:48:22.671782 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.671764 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg4hs\" (UniqueName: \"kubernetes.io/projected/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-kube-api-access-qg4hs\") pod \"maas-api-revoke-sub-fef94b89-94kk6\" (UID: \"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6" Nov 02 20:48:22.671892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.671869 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fzcb\" (UniqueName: \"kubernetes.io/projected/be58e326-8b22-4a60-a9fb-0a4546a54ae7-kube-api-access-2fzcb\") pod \"maas-api-revoke-sub-6e3fffe3-vdv8t\" (UID: \"be58e326-8b22-4a60-a9fb-0a4546a54ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" Nov 02 20:48:22.708816 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.708798 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" Nov 02 20:48:22.763360 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:22.763328 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6" Nov 02 20:48:23.077328 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:23.077295 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t"] Nov 02 20:48:23.078279 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:48:23.078217 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe58e326_8b22_4a60_a9fb_0a4546a54ae7.slice/crio-9303185fef1e0df63e49b017fefe0dd5a61369902426ad3a2e81a83e3a3c5462 WatchSource:0}: Error finding container 9303185fef1e0df63e49b017fefe0dd5a61369902426ad3a2e81a83e3a3c5462: Status 404 returned error can't find the container with id 9303185fef1e0df63e49b017fefe0dd5a61369902426ad3a2e81a83e3a3c5462 Nov 02 20:48:23.171227 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:23.171200 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6"] Nov 02 20:48:23.171864 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:48:23.171836 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6d6ca0_4f04_4ee2_b6ad_7b5e4961bc64.slice/crio-25223d26550ea816711c6a41850826114eb89e6041ce2144dbbaffcea00027a7 WatchSource:0}: Error finding container 25223d26550ea816711c6a41850826114eb89e6041ce2144dbbaffcea00027a7: Status 404 returned error can't find the container with id 25223d26550ea816711c6a41850826114eb89e6041ce2144dbbaffcea00027a7 Nov 02 20:48:23.323040 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:23.323005 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:23.363695 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:23.363666 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" event={"ID":"be58e326-8b22-4a60-a9fb-0a4546a54ae7","Type":"ContainerStarted","Data":"668f338dd66a149e00e582d53f935c5cb6a2d7046394b3620eee79cbadcfa3c9"} Nov 02 20:48:23.363821 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:23.363729 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" event={"ID":"be58e326-8b22-4a60-a9fb-0a4546a54ae7","Type":"ContainerStarted","Data":"9303185fef1e0df63e49b017fefe0dd5a61369902426ad3a2e81a83e3a3c5462"} Nov 02 20:48:23.365048 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:23.365016 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6" event={"ID":"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64","Type":"ContainerStarted","Data":"25223d26550ea816711c6a41850826114eb89e6041ce2144dbbaffcea00027a7"} Nov 02 20:48:23.474618 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:23.474576 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" podStartSLOduration=1.474561839 podStartE2EDuration="1.474561839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:48:23.472019074 +0000 UTC m=+2031.761675602" watchObservedRunningTime="2026-11-02 20:48:23.474561839 +0000 UTC m=+2031.764218367" Nov 02 20:48:24.166323 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:24.166289 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:24.383204 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:24.383157 4971 generic.go:356] "Generic (PLEG): container finished" podID="be58e326-8b22-4a60-a9fb-0a4546a54ae7" containerID="668f338dd66a149e00e582d53f935c5cb6a2d7046394b3620eee79cbadcfa3c9" exitCode=0 Nov 02 20:48:24.383382 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:24.383230 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" event={"ID":"be58e326-8b22-4a60-a9fb-0a4546a54ae7","Type":"ContainerDied","Data":"668f338dd66a149e00e582d53f935c5cb6a2d7046394b3620eee79cbadcfa3c9"} Nov 02 20:48:24.385192 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:24.385167 4971 generic.go:356] "Generic (PLEG): container finished" podID="ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64" containerID="9c37f95a70394bdc8104ac83416009a4902148e7f816880dd70d713d28820d45" exitCode=0 Nov 02 20:48:24.385332 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:24.385249 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6" event={"ID":"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64","Type":"ContainerDied","Data":"9c37f95a70394bdc8104ac83416009a4902148e7f816880dd70d713d28820d45"} Nov 02 20:48:24.419779 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:24.419752 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:25.555329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.555270 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" Nov 02 20:48:25.560497 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.560458 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m"] Nov 02 20:48:25.561264 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.560704 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" podUID="35418b46-29f4-48ba-a0ac-8ca3243c4152" containerName="istio-proxy" containerID="cri-o://2247131d43df52af6933e053c9c7e134126fa355d931794b1acca7310c875591" gracePeriod=30 Nov 02 20:48:25.600492 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.600468 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6" Nov 02 20:48:25.648357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.648291 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be58e326-8b22-4a60-a9fb-0a4546a54ae7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be58e326-8b22-4a60-a9fb-0a4546a54ae7-maas-api-openshift-ca\") pod \"be58e326-8b22-4a60-a9fb-0a4546a54ae7\" (UID: \"be58e326-8b22-4a60-a9fb-0a4546a54ae7\") " Nov 02 20:48:25.648513 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.648372 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-maas-api-xks-ca\") pod \"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64\" (UID: \"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64\") " Nov 02 20:48:25.648513 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.648413 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-kube-api-access-qg4hs\" (UniqueName: \"kubernetes.io/projected/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-kube-api-access-qg4hs\") pod \"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64\" (UID: \"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64\") " Nov 02 20:48:25.648513 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.648448 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-maas-api-openshift-ca\") pod \"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64\" (UID: \"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64\") " Nov 02 20:48:25.648704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.648518 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be58e326-8b22-4a60-a9fb-0a4546a54ae7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be58e326-8b22-4a60-a9fb-0a4546a54ae7-maas-api-xks-ca\") pod \"be58e326-8b22-4a60-a9fb-0a4546a54ae7\" (UID: \"be58e326-8b22-4a60-a9fb-0a4546a54ae7\") " Nov 02 20:48:25.648704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.648557 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be58e326-8b22-4a60-a9fb-0a4546a54ae7-kube-api-access-2fzcb\" (UniqueName: \"kubernetes.io/projected/be58e326-8b22-4a60-a9fb-0a4546a54ae7-kube-api-access-2fzcb\") pod \"be58e326-8b22-4a60-a9fb-0a4546a54ae7\" (UID: \"be58e326-8b22-4a60-a9fb-0a4546a54ae7\") " Nov 02 20:48:25.648820 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.648670 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be58e326-8b22-4a60-a9fb-0a4546a54ae7-maas-api-openshift-ca" pod "be58e326-8b22-4a60-a9fb-0a4546a54ae7" (UID: "be58e326-8b22-4a60-a9fb-0a4546a54ae7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:25.648902 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.648875 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-maas-api-openshift-ca" pod "ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64" (UID: "ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:25.649062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.649037 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be58e326-8b22-4a60-a9fb-0a4546a54ae7-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:25.649161 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.649067 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:25.651064 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.651038 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-maas-api-xks-ca" pod "ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64" (UID: "ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:25.651188 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.651112 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-kube-api-access-qg4hs" pod "ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64" (UID: "ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64"). InnerVolumeSpecName "kube-api-access-qg4hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:25.651349 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.651328 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be58e326-8b22-4a60-a9fb-0a4546a54ae7-maas-api-xks-ca" pod "be58e326-8b22-4a60-a9fb-0a4546a54ae7" (UID: "be58e326-8b22-4a60-a9fb-0a4546a54ae7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:25.651412 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.651373 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be58e326-8b22-4a60-a9fb-0a4546a54ae7-kube-api-access-2fzcb" pod "be58e326-8b22-4a60-a9fb-0a4546a54ae7" (UID: "be58e326-8b22-4a60-a9fb-0a4546a54ae7"). InnerVolumeSpecName "kube-api-access-2fzcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:25.749867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.749835 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be58e326-8b22-4a60-a9fb-0a4546a54ae7-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:25.749867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.749863 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fzcb\" (UniqueName: \"kubernetes.io/projected/be58e326-8b22-4a60-a9fb-0a4546a54ae7-kube-api-access-2fzcb\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:25.749867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.749874 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:25.750132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.749883 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg4hs\" (UniqueName: \"kubernetes.io/projected/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64-kube-api-access-qg4hs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:25.944601 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:25.944526 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:26.400042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:26.400015 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" Nov 02 20:48:26.400222 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:26.400056 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t" event={"ID":"be58e326-8b22-4a60-a9fb-0a4546a54ae7","Type":"ContainerDied","Data":"9303185fef1e0df63e49b017fefe0dd5a61369902426ad3a2e81a83e3a3c5462"} Nov 02 20:48:26.400222 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:26.400126 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9303185fef1e0df63e49b017fefe0dd5a61369902426ad3a2e81a83e3a3c5462" Nov 02 20:48:26.401623 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:26.401604 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6" Nov 02 20:48:26.401752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:26.401627 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6" event={"ID":"ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64","Type":"ContainerDied","Data":"25223d26550ea816711c6a41850826114eb89e6041ce2144dbbaffcea00027a7"} Nov 02 20:48:26.401752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:26.401657 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25223d26550ea816711c6a41850826114eb89e6041ce2144dbbaffcea00027a7" Nov 02 20:48:27.085714 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:27.085684 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8"] Nov 02 20:48:27.195735 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:27.195701 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf"] Nov 02 20:48:27.555359 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:27.555324 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:27.803463 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:27.803433 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:28.265232 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:28.265199 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:30.823053 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.823022 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:48:30.906946 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.906909 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-podinfo\") pod \"35418b46-29f4-48ba-a0ac-8ca3243c4152\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " Nov 02 20:48:30.907113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.907036 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-data\") pod \"35418b46-29f4-48ba-a0ac-8ca3243c4152\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " Nov 02 20:48:30.907113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.907100 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-token\" (UniqueName: \"kubernetes.io/projected/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-token\") pod \"35418b46-29f4-48ba-a0ac-8ca3243c4152\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " Nov 02 20:48:30.907245 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.907136 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-workload-socket\") pod \"35418b46-29f4-48ba-a0ac-8ca3243c4152\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " Nov 02 20:48:30.907245 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.907172 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-envoy\") pod \"35418b46-29f4-48ba-a0ac-8ca3243c4152\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " Nov 02 20:48:30.907245 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.907204 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-workload-certs\") pod \"35418b46-29f4-48ba-a0ac-8ca3243c4152\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " Nov 02 20:48:30.907413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.907271 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35418b46-29f4-48ba-a0ac-8ca3243c4152-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35418b46-29f4-48ba-a0ac-8ca3243c4152-istiod-ca-cert\") pod \"35418b46-29f4-48ba-a0ac-8ca3243c4152\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " Nov 02 20:48:30.907413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.907308 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35418b46-29f4-48ba-a0ac-8ca3243c4152-kube-api-access-4hjb5\" (UniqueName: \"kubernetes.io/projected/35418b46-29f4-48ba-a0ac-8ca3243c4152-kube-api-access-4hjb5\") pod \"35418b46-29f4-48ba-a0ac-8ca3243c4152\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " Nov 02 20:48:30.907413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.907356 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-data" pod "35418b46-29f4-48ba-a0ac-8ca3243c4152" (UID: "35418b46-29f4-48ba-a0ac-8ca3243c4152"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:30.907413 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.907380 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-credential-socket\") pod \"35418b46-29f4-48ba-a0ac-8ca3243c4152\" (UID: \"35418b46-29f4-48ba-a0ac-8ca3243c4152\") " Nov 02 20:48:30.907622 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.907487 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-workload-socket" pod "35418b46-29f4-48ba-a0ac-8ca3243c4152" (UID: "35418b46-29f4-48ba-a0ac-8ca3243c4152"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:30.907904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.907760 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-credential-socket" pod "35418b46-29f4-48ba-a0ac-8ca3243c4152" (UID: "35418b46-29f4-48ba-a0ac-8ca3243c4152"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:30.907904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.907837 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:30.907904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.907858 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:30.907904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.907875 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35418b46-29f4-48ba-a0ac-8ca3243c4152-istiod-ca-cert" pod "35418b46-29f4-48ba-a0ac-8ca3243c4152" (UID: "35418b46-29f4-48ba-a0ac-8ca3243c4152"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:30.908233 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.908128 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-workload-certs" pod "35418b46-29f4-48ba-a0ac-8ca3243c4152" (UID: "35418b46-29f4-48ba-a0ac-8ca3243c4152"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:30.909479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.909451 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-token" pod "35418b46-29f4-48ba-a0ac-8ca3243c4152" (UID: "35418b46-29f4-48ba-a0ac-8ca3243c4152"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:30.909588 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.909526 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-podinfo" pod "35418b46-29f4-48ba-a0ac-8ca3243c4152" (UID: "35418b46-29f4-48ba-a0ac-8ca3243c4152"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:48:30.909639 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.909618 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35418b46-29f4-48ba-a0ac-8ca3243c4152-kube-api-access-4hjb5" pod "35418b46-29f4-48ba-a0ac-8ca3243c4152" (UID: "35418b46-29f4-48ba-a0ac-8ca3243c4152"). InnerVolumeSpecName "kube-api-access-4hjb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:30.910027 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:30.909999 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-envoy" pod "35418b46-29f4-48ba-a0ac-8ca3243c4152" (UID: "35418b46-29f4-48ba-a0ac-8ca3243c4152"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:31.014634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.014504 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:31.014634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.014539 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:31.014634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.014553 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:31.014634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.014566 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:31.014634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.014579 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35418b46-29f4-48ba-a0ac-8ca3243c4152-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:31.014634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.014592 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35418b46-29f4-48ba-a0ac-8ca3243c4152-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:31.014634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.014605 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hjb5\" (UniqueName: \"kubernetes.io/projected/35418b46-29f4-48ba-a0ac-8ca3243c4152-kube-api-access-4hjb5\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:31.438837 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.438799 4971 generic.go:356] "Generic (PLEG): container finished" podID="35418b46-29f4-48ba-a0ac-8ca3243c4152" containerID="2247131d43df52af6933e053c9c7e134126fa355d931794b1acca7310c875591" exitCode=0 Nov 02 20:48:31.439030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.438846 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" Nov 02 20:48:31.439030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.438859 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" event={"ID":"35418b46-29f4-48ba-a0ac-8ca3243c4152","Type":"ContainerDied","Data":"2247131d43df52af6933e053c9c7e134126fa355d931794b1acca7310c875591"} Nov 02 20:48:31.439030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.438910 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m" event={"ID":"35418b46-29f4-48ba-a0ac-8ca3243c4152","Type":"ContainerDied","Data":"df24dc8952f16bf6c9458c85f676be2e367e62ee69e94b410722950266e9cec3"} Nov 02 20:48:31.439030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.438939 4971 scope.go:122] "RemoveContainer" containerID="2247131d43df52af6933e053c9c7e134126fa355d931794b1acca7310c875591" Nov 02 20:48:31.450434 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.450411 4971 scope.go:122] "RemoveContainer" containerID="2247131d43df52af6933e053c9c7e134126fa355d931794b1acca7310c875591" Nov 02 20:48:31.450731 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:31.450709 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2247131d43df52af6933e053c9c7e134126fa355d931794b1acca7310c875591\": container with ID starting with 2247131d43df52af6933e053c9c7e134126fa355d931794b1acca7310c875591 not found: ID does not exist" containerID="2247131d43df52af6933e053c9c7e134126fa355d931794b1acca7310c875591" Nov 02 20:48:31.450801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.450748 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2247131d43df52af6933e053c9c7e134126fa355d931794b1acca7310c875591"} err="failed to get container status \"2247131d43df52af6933e053c9c7e134126fa355d931794b1acca7310c875591\": rpc error: code = NotFound desc = could not find container \"2247131d43df52af6933e053c9c7e134126fa355d931794b1acca7310c875591\": container with ID starting with 2247131d43df52af6933e053c9c7e134126fa355d931794b1acca7310c875591 not found: ID does not exist" Nov 02 20:48:31.596314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.596270 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m"] Nov 02 20:48:31.642174 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.642145 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vxc8m"] Nov 02 20:48:31.650816 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.650783 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:31.786318 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.786253 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" podUID="a29a43c7-6192-4151-9c21-bcedab45ba8c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:48:31.786318 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.786272 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" podUID="ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:48:31.867993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.867969 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35418b46-29f4-48ba-a0ac-8ca3243c4152" path="/var/lib/kubelet/pods/35418b46-29f4-48ba-a0ac-8ca3243c4152/volumes" Nov 02 20:48:31.894888 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:31.894864 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:32.086220 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.086137 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" podUID="9525e1da-30ce-46db-8588-8f67c46da099" containerName="payload-processing" containerID="cri-o://9cb6331398dac41ce382f32d59116dff2bedc2d1e5b686fb67e598eab579713b" gracePeriod=25 Nov 02 20:48:32.196155 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.196120 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" podUID="06a7d33d-32b2-4efe-8373-40491c26931d" containerName="payload-pre-processing" containerID="cri-o://c028c544307efc9c3e74c5c5167e9dab37a5a645013f8e67a2fc39856f1ca5ec" gracePeriod=25 Nov 02 20:48:32.380713 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.380682 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" Nov 02 20:48:32.448879 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.448839 4971 generic.go:356] "Generic (PLEG): container finished" podID="06a7d33d-32b2-4efe-8373-40491c26931d" containerID="c028c544307efc9c3e74c5c5167e9dab37a5a645013f8e67a2fc39856f1ca5ec" exitCode=0 Nov 02 20:48:32.449067 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.449045 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" event={"ID":"06a7d33d-32b2-4efe-8373-40491c26931d","Type":"ContainerDied","Data":"c028c544307efc9c3e74c5c5167e9dab37a5a645013f8e67a2fc39856f1ca5ec"} Nov 02 20:48:32.455042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.451198 4971 generic.go:356] "Generic (PLEG): container finished" podID="9525e1da-30ce-46db-8588-8f67c46da099" containerID="9cb6331398dac41ce382f32d59116dff2bedc2d1e5b686fb67e598eab579713b" exitCode=0 Nov 02 20:48:32.455998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.455266 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" event={"ID":"9525e1da-30ce-46db-8588-8f67c46da099","Type":"ContainerDied","Data":"9cb6331398dac41ce382f32d59116dff2bedc2d1e5b686fb67e598eab579713b"} Nov 02 20:48:32.455998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.455331 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" event={"ID":"9525e1da-30ce-46db-8588-8f67c46da099","Type":"ContainerDied","Data":"01368f0211d8d4e272f21c545d4e6ef28cc5ad491fc7cdc403ab33ed99325ac8"} Nov 02 20:48:32.455998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.455363 4971 scope.go:122] "RemoveContainer" containerID="9cb6331398dac41ce382f32d59116dff2bedc2d1e5b686fb67e598eab579713b" Nov 02 20:48:32.455998 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.455687 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8" Nov 02 20:48:32.456528 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.456453 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:32.475787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.475755 4971 scope.go:122] "RemoveContainer" containerID="9cb6331398dac41ce382f32d59116dff2bedc2d1e5b686fb67e598eab579713b" Nov 02 20:48:32.476217 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:32.476185 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cb6331398dac41ce382f32d59116dff2bedc2d1e5b686fb67e598eab579713b\": container with ID starting with 9cb6331398dac41ce382f32d59116dff2bedc2d1e5b686fb67e598eab579713b not found: ID does not exist" containerID="9cb6331398dac41ce382f32d59116dff2bedc2d1e5b686fb67e598eab579713b" Nov 02 20:48:32.476334 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.476230 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb6331398dac41ce382f32d59116dff2bedc2d1e5b686fb67e598eab579713b"} err="failed to get container status \"9cb6331398dac41ce382f32d59116dff2bedc2d1e5b686fb67e598eab579713b\": rpc error: code = NotFound desc = could not find container \"9cb6331398dac41ce382f32d59116dff2bedc2d1e5b686fb67e598eab579713b\": container with ID starting with 9cb6331398dac41ce382f32d59116dff2bedc2d1e5b686fb67e598eab579713b not found: ID does not exist" Nov 02 20:48:32.499859 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.499827 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" Nov 02 20:48:32.528925 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.528889 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9525e1da-30ce-46db-8588-8f67c46da099-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9525e1da-30ce-46db-8588-8f67c46da099-plugins-config-volume\") pod \"9525e1da-30ce-46db-8588-8f67c46da099\" (UID: \"9525e1da-30ce-46db-8588-8f67c46da099\") " Nov 02 20:48:32.529104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.528968 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9525e1da-30ce-46db-8588-8f67c46da099-kube-api-access-n254d\" (UniqueName: \"kubernetes.io/projected/9525e1da-30ce-46db-8588-8f67c46da099-kube-api-access-n254d\") pod \"9525e1da-30ce-46db-8588-8f67c46da099\" (UID: \"9525e1da-30ce-46db-8588-8f67c46da099\") " Nov 02 20:48:32.529338 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.529308 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9525e1da-30ce-46db-8588-8f67c46da099-plugins-config-volume" pod "9525e1da-30ce-46db-8588-8f67c46da099" (UID: "9525e1da-30ce-46db-8588-8f67c46da099"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:32.529497 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.529478 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9525e1da-30ce-46db-8588-8f67c46da099-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:32.531478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.531451 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9525e1da-30ce-46db-8588-8f67c46da099-kube-api-access-n254d" pod "9525e1da-30ce-46db-8588-8f67c46da099" (UID: "9525e1da-30ce-46db-8588-8f67c46da099"). InnerVolumeSpecName "kube-api-access-n254d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:32.639410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.638831 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a7d33d-32b2-4efe-8373-40491c26931d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06a7d33d-32b2-4efe-8373-40491c26931d-plugins-config-volume\") pod \"06a7d33d-32b2-4efe-8373-40491c26931d\" (UID: \"06a7d33d-32b2-4efe-8373-40491c26931d\") " Nov 02 20:48:32.639410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.638961 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a7d33d-32b2-4efe-8373-40491c26931d-kube-api-access-vrbdz\" (UniqueName: \"kubernetes.io/projected/06a7d33d-32b2-4efe-8373-40491c26931d-kube-api-access-vrbdz\") pod \"06a7d33d-32b2-4efe-8373-40491c26931d\" (UID: \"06a7d33d-32b2-4efe-8373-40491c26931d\") " Nov 02 20:48:32.639410 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.639292 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a7d33d-32b2-4efe-8373-40491c26931d-plugins-config-volume" pod "06a7d33d-32b2-4efe-8373-40491c26931d" (UID: "06a7d33d-32b2-4efe-8373-40491c26931d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:32.639881 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.639455 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06a7d33d-32b2-4efe-8373-40491c26931d-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:32.639881 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.639479 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n254d\" (UniqueName: \"kubernetes.io/projected/9525e1da-30ce-46db-8588-8f67c46da099-kube-api-access-n254d\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:32.642214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.642184 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a7d33d-32b2-4efe-8373-40491c26931d-kube-api-access-vrbdz" pod "06a7d33d-32b2-4efe-8373-40491c26931d" (UID: "06a7d33d-32b2-4efe-8373-40491c26931d"). InnerVolumeSpecName "kube-api-access-vrbdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:32.741173 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.741143 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrbdz\" (UniqueName: \"kubernetes.io/projected/06a7d33d-32b2-4efe-8373-40491c26931d-kube-api-access-vrbdz\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:32.890679 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.890602 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8"] Nov 02 20:48:32.896890 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:32.896859 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-64ff15-69b8d9489d-cr7m8"] Nov 02 20:48:33.338842 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:33.338766 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:48:33.463344 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.463308 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" Nov 02 20:48:33.463344 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.463329 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf" event={"ID":"06a7d33d-32b2-4efe-8373-40491c26931d","Type":"ContainerDied","Data":"1cb152ba13ae7d306a6d31b2ebff6a439c78f1ad74a305716d2dd86df08606cf"} Nov 02 20:48:33.463595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.463395 4971 scope.go:122] "RemoveContainer" containerID="c028c544307efc9c3e74c5c5167e9dab37a5a645013f8e67a2fc39856f1ca5ec" Nov 02 20:48:33.468595 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.468566 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:33.511015 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.510993 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6"] Nov 02 20:48:33.512118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.512097 4971 cpu_manager.go:414] "containerMap: removing container" podUID="06a7d33d-32b2-4efe-8373-40491c26931d" containerName="payload-pre-processing" Nov 02 20:48:33.512118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.512117 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a7d33d-32b2-4efe-8373-40491c26931d" containerName="payload-pre-processing" Nov 02 20:48:33.512272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.512132 4971 cpu_manager.go:414] "containerMap: removing container" podUID="be58e326-8b22-4a60-a9fb-0a4546a54ae7" containerName="revoke-keys" Nov 02 20:48:33.512272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.512138 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be58e326-8b22-4a60-a9fb-0a4546a54ae7" containerName="revoke-keys" Nov 02 20:48:33.512272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.512160 4971 cpu_manager.go:414] "containerMap: removing container" podUID="9525e1da-30ce-46db-8588-8f67c46da099" containerName="payload-processing" Nov 02 20:48:33.512272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.512167 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9525e1da-30ce-46db-8588-8f67c46da099" containerName="payload-processing" Nov 02 20:48:33.512272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.512183 4971 cpu_manager.go:414] "containerMap: removing container" podUID="35418b46-29f4-48ba-a0ac-8ca3243c4152" containerName="istio-proxy" Nov 02 20:48:33.512272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.512191 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35418b46-29f4-48ba-a0ac-8ca3243c4152" containerName="istio-proxy" Nov 02 20:48:33.512272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.512216 4971 cpu_manager.go:414] "containerMap: removing container" podUID="ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64" containerName="revoke-keys" Nov 02 20:48:33.512272 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.512220 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64" containerName="revoke-keys" Nov 02 20:48:33.512533 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.512325 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="9525e1da-30ce-46db-8588-8f67c46da099" containerName="payload-processing" Nov 02 20:48:33.512533 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.512336 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="35418b46-29f4-48ba-a0ac-8ca3243c4152" containerName="istio-proxy" Nov 02 20:48:33.512533 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.512346 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64" containerName="revoke-keys" Nov 02 20:48:33.512533 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.512354 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a7d33d-32b2-4efe-8373-40491c26931d" containerName="payload-pre-processing" Nov 02 20:48:33.512533 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.512361 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="be58e326-8b22-4a60-a9fb-0a4546a54ae7" containerName="revoke-keys" Nov 02 20:48:33.517236 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.517219 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6" Nov 02 20:48:33.565013 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.564986 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn"] Nov 02 20:48:33.568963 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.568938 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:48:33.569109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.569076 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:48:33.569109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.569104 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn" Nov 02 20:48:33.572846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.572825 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6"] Nov 02 20:48:33.621127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.621102 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf"] Nov 02 20:48:33.628070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.628030 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn"] Nov 02 20:48:33.632983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.632959 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-64ff15-bd5457bd7-kfspf"] Nov 02 20:48:33.652432 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.652405 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e18bb6d4-tzlz6\" (UID: \"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6" Nov 02 20:48:33.652559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.652464 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbtdn\" (UniqueName: \"kubernetes.io/projected/dc54304f-438e-4dd9-8338-16b67bab73b7-kube-api-access-nbtdn\") pod \"maas-api-revoke-sub-257e022b-pzmnn\" (UID: \"dc54304f-438e-4dd9-8338-16b67bab73b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn" Nov 02 20:48:33.652559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.652545 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc54304f-438e-4dd9-8338-16b67bab73b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-257e022b-pzmnn\" (UID: \"dc54304f-438e-4dd9-8338-16b67bab73b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn" Nov 02 20:48:33.652839 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.652713 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n667\" (UniqueName: \"kubernetes.io/projected/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-kube-api-access-9n667\") pod \"maas-api-revoke-sub-e18bb6d4-tzlz6\" (UID: \"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6" Nov 02 20:48:33.652839 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.652765 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e18bb6d4-tzlz6\" (UID: \"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6" Nov 02 20:48:33.652955 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.652864 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc54304f-438e-4dd9-8338-16b67bab73b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-257e022b-pzmnn\" (UID: \"dc54304f-438e-4dd9-8338-16b67bab73b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn" Nov 02 20:48:33.754501 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.754471 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n667\" (UniqueName: \"kubernetes.io/projected/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-kube-api-access-9n667\") pod \"maas-api-revoke-sub-e18bb6d4-tzlz6\" (UID: \"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6" Nov 02 20:48:33.754653 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.754526 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e18bb6d4-tzlz6\" (UID: \"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6" Nov 02 20:48:33.754653 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.754559 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc54304f-438e-4dd9-8338-16b67bab73b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-257e022b-pzmnn\" (UID: \"dc54304f-438e-4dd9-8338-16b67bab73b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn" Nov 02 20:48:33.754653 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.754593 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e18bb6d4-tzlz6\" (UID: \"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6" Nov 02 20:48:33.754653 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.754616 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbtdn\" (UniqueName: \"kubernetes.io/projected/dc54304f-438e-4dd9-8338-16b67bab73b7-kube-api-access-nbtdn\") pod \"maas-api-revoke-sub-257e022b-pzmnn\" (UID: \"dc54304f-438e-4dd9-8338-16b67bab73b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn" Nov 02 20:48:33.754653 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.754632 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc54304f-438e-4dd9-8338-16b67bab73b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-257e022b-pzmnn\" (UID: \"dc54304f-438e-4dd9-8338-16b67bab73b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn" Nov 02 20:48:33.755362 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.755337 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e18bb6d4-tzlz6\" (UID: \"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6" Nov 02 20:48:33.755614 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.755589 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc54304f-438e-4dd9-8338-16b67bab73b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-257e022b-pzmnn\" (UID: \"dc54304f-438e-4dd9-8338-16b67bab73b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn" Nov 02 20:48:33.757167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.757147 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc54304f-438e-4dd9-8338-16b67bab73b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-257e022b-pzmnn\" (UID: \"dc54304f-438e-4dd9-8338-16b67bab73b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn" Nov 02 20:48:33.757265 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.757209 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e18bb6d4-tzlz6\" (UID: \"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6" Nov 02 20:48:33.812328 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.812306 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n667\" (UniqueName: \"kubernetes.io/projected/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-kube-api-access-9n667\") pod \"maas-api-revoke-sub-e18bb6d4-tzlz6\" (UID: \"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6" Nov 02 20:48:33.812545 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.812526 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbtdn\" (UniqueName: \"kubernetes.io/projected/dc54304f-438e-4dd9-8338-16b67bab73b7-kube-api-access-nbtdn\") pod \"maas-api-revoke-sub-257e022b-pzmnn\" (UID: \"dc54304f-438e-4dd9-8338-16b67bab73b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn" Nov 02 20:48:33.830177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.830154 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6" Nov 02 20:48:33.867525 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.867492 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a7d33d-32b2-4efe-8373-40491c26931d" path="/var/lib/kubelet/pods/06a7d33d-32b2-4efe-8373-40491c26931d/volumes" Nov 02 20:48:33.868122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.868103 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9525e1da-30ce-46db-8588-8f67c46da099" path="/var/lib/kubelet/pods/9525e1da-30ce-46db-8588-8f67c46da099/volumes" Nov 02 20:48:33.881253 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.881204 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn" Nov 02 20:48:33.959257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:33.955013 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:34.205323 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:48:34.205287 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f508d04_a0a8_4e9c_bc78_cf2e4c6211d1.slice/crio-acc13436068dc66dc14a17ae31a20c1a4aba91417ba65901e3f706b508d4205e WatchSource:0}: Error finding container acc13436068dc66dc14a17ae31a20c1a4aba91417ba65901e3f706b508d4205e: Status 404 returned error can't find the container with id acc13436068dc66dc14a17ae31a20c1a4aba91417ba65901e3f706b508d4205e Nov 02 20:48:34.205626 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:34.205592 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6"] Nov 02 20:48:34.228435 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:34.228411 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:34.298200 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:34.298165 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn"] Nov 02 20:48:34.298653 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:48:34.298627 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc54304f_438e_4dd9_8338_16b67bab73b7.slice/crio-41b300100ee09af59927b69df5802f545c0576918931d79b86e56752246dc7b6 WatchSource:0}: Error finding container 41b300100ee09af59927b69df5802f545c0576918931d79b86e56752246dc7b6: Status 404 returned error can't find the container with id 41b300100ee09af59927b69df5802f545c0576918931d79b86e56752246dc7b6 Nov 02 20:48:34.355784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:34.355760 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885"] Nov 02 20:48:34.364240 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:34.364216 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4166b0-rh885"] Nov 02 20:48:34.473164 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:34.473068 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn" event={"ID":"dc54304f-438e-4dd9-8338-16b67bab73b7","Type":"ContainerStarted","Data":"41b300100ee09af59927b69df5802f545c0576918931d79b86e56752246dc7b6"} Nov 02 20:48:34.475313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:34.475290 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6" event={"ID":"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1","Type":"ContainerStarted","Data":"acc13436068dc66dc14a17ae31a20c1a4aba91417ba65901e3f706b508d4205e"} Nov 02 20:48:34.480252 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:34.480230 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:34.862625 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:34.862595 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:35.483753 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.483670 4971 generic.go:356] "Generic (PLEG): container finished" podID="dc54304f-438e-4dd9-8338-16b67bab73b7" containerID="f088be4e06509dcf8a7a5f7e25a8834f33d8c14c5c0a10d9c82ce58aa6d4cf86" exitCode=0 Nov 02 20:48:35.484248 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.483802 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn" event={"ID":"dc54304f-438e-4dd9-8338-16b67bab73b7","Type":"ContainerDied","Data":"f088be4e06509dcf8a7a5f7e25a8834f33d8c14c5c0a10d9c82ce58aa6d4cf86"} Nov 02 20:48:35.485677 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.485652 4971 generic.go:356] "Generic (PLEG): container finished" podID="9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1" containerID="648aadaf6c656e2ff3177a43dd6f74d819c85ed1e1be551487cd53422aff3524" exitCode=0 Nov 02 20:48:35.485793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.485683 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6" event={"ID":"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1","Type":"ContainerDied","Data":"648aadaf6c656e2ff3177a43dd6f74d819c85ed1e1be551487cd53422aff3524"} Nov 02 20:48:35.643172 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.643142 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" Nov 02 20:48:35.745562 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.745539 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" Nov 02 20:48:35.776911 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.776876 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29a43c7-6192-4151-9c21-bcedab45ba8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29a43c7-6192-4151-9c21-bcedab45ba8c-plugins-config-volume\") pod \"a29a43c7-6192-4151-9c21-bcedab45ba8c\" (UID: \"a29a43c7-6192-4151-9c21-bcedab45ba8c\") " Nov 02 20:48:35.777050 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.776983 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29a43c7-6192-4151-9c21-bcedab45ba8c-kube-api-access-cwb6t\" (UniqueName: \"kubernetes.io/projected/a29a43c7-6192-4151-9c21-bcedab45ba8c-kube-api-access-cwb6t\") pod \"a29a43c7-6192-4151-9c21-bcedab45ba8c\" (UID: \"a29a43c7-6192-4151-9c21-bcedab45ba8c\") " Nov 02 20:48:35.777333 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.777310 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29a43c7-6192-4151-9c21-bcedab45ba8c-plugins-config-volume" pod "a29a43c7-6192-4151-9c21-bcedab45ba8c" (UID: "a29a43c7-6192-4151-9c21-bcedab45ba8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:35.777537 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.777525 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29a43c7-6192-4151-9c21-bcedab45ba8c-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:35.779231 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.779212 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29a43c7-6192-4151-9c21-bcedab45ba8c-kube-api-access-cwb6t" pod "a29a43c7-6192-4151-9c21-bcedab45ba8c" (UID: "a29a43c7-6192-4151-9c21-bcedab45ba8c"). InnerVolumeSpecName "kube-api-access-cwb6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:35.868728 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.868697 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2f082b-62a4-4bde-80b8-399f286abd57" path="/var/lib/kubelet/pods/2d2f082b-62a4-4bde-80b8-399f286abd57/volumes" Nov 02 20:48:35.877816 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.877787 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48-kube-api-access-79b7b\" (UniqueName: \"kubernetes.io/projected/ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48-kube-api-access-79b7b\") pod \"ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48\" (UID: \"ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48\") " Nov 02 20:48:35.877934 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.877908 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48-plugins-config-volume\") pod \"ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48\" (UID: \"ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48\") " Nov 02 20:48:35.878359 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.878329 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48-plugins-config-volume" pod "ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48" (UID: "ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:35.878489 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.878338 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwb6t\" (UniqueName: \"kubernetes.io/projected/a29a43c7-6192-4151-9c21-bcedab45ba8c-kube-api-access-cwb6t\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:35.880260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.880240 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48-kube-api-access-79b7b" pod "ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48" (UID: "ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48"). InnerVolumeSpecName "kube-api-access-79b7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:35.979042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.979013 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:35.979042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:35.979039 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79b7b\" (UniqueName: \"kubernetes.io/projected/ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48-kube-api-access-79b7b\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:36.494256 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.494172 4971 generic.go:356] "Generic (PLEG): container finished" podID="a29a43c7-6192-4151-9c21-bcedab45ba8c" containerID="47065991283d41332d9dd31f44e4252ec6be2c9b4c15f51df5d8f9d2ef3da751" exitCode=0 Nov 02 20:48:36.494256 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.494240 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" Nov 02 20:48:36.494734 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.494254 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" event={"ID":"a29a43c7-6192-4151-9c21-bcedab45ba8c","Type":"ContainerDied","Data":"47065991283d41332d9dd31f44e4252ec6be2c9b4c15f51df5d8f9d2ef3da751"} Nov 02 20:48:36.494734 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.494312 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t" event={"ID":"a29a43c7-6192-4151-9c21-bcedab45ba8c","Type":"ContainerDied","Data":"41ca70ad9f3624bfd43f9c243ce1e6781ded7f7b11e6924f8b64f21c8efb4afc"} Nov 02 20:48:36.494734 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.494334 4971 scope.go:122] "RemoveContainer" containerID="47065991283d41332d9dd31f44e4252ec6be2c9b4c15f51df5d8f9d2ef3da751" Nov 02 20:48:36.495905 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.495879 4971 generic.go:356] "Generic (PLEG): container finished" podID="ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48" containerID="da848c22a288dc7a52c88904ff7c4c9dc75a9b6b7556a00424b59c76eaa85466" exitCode=0 Nov 02 20:48:36.496040 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.495930 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" Nov 02 20:48:36.496040 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.495946 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" event={"ID":"ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48","Type":"ContainerDied","Data":"da848c22a288dc7a52c88904ff7c4c9dc75a9b6b7556a00424b59c76eaa85466"} Nov 02 20:48:36.496040 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.495991 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558" event={"ID":"ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48","Type":"ContainerDied","Data":"623f63bd49d7535e012e6423b6c137163e40b7f8ad62837ca4d998860907616e"} Nov 02 20:48:36.507582 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.507554 4971 scope.go:122] "RemoveContainer" containerID="47065991283d41332d9dd31f44e4252ec6be2c9b4c15f51df5d8f9d2ef3da751" Nov 02 20:48:36.507957 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:36.507933 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47065991283d41332d9dd31f44e4252ec6be2c9b4c15f51df5d8f9d2ef3da751\": container with ID starting with 47065991283d41332d9dd31f44e4252ec6be2c9b4c15f51df5d8f9d2ef3da751 not found: ID does not exist" containerID="47065991283d41332d9dd31f44e4252ec6be2c9b4c15f51df5d8f9d2ef3da751" Nov 02 20:48:36.508056 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.507964 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47065991283d41332d9dd31f44e4252ec6be2c9b4c15f51df5d8f9d2ef3da751"} err="failed to get container status \"47065991283d41332d9dd31f44e4252ec6be2c9b4c15f51df5d8f9d2ef3da751\": rpc error: code = NotFound desc = could not find container \"47065991283d41332d9dd31f44e4252ec6be2c9b4c15f51df5d8f9d2ef3da751\": container with ID starting with 47065991283d41332d9dd31f44e4252ec6be2c9b4c15f51df5d8f9d2ef3da751 not found: ID does not exist" Nov 02 20:48:36.508056 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.507982 4971 scope.go:122] "RemoveContainer" containerID="da848c22a288dc7a52c88904ff7c4c9dc75a9b6b7556a00424b59c76eaa85466" Nov 02 20:48:36.520195 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.520176 4971 scope.go:122] "RemoveContainer" containerID="da848c22a288dc7a52c88904ff7c4c9dc75a9b6b7556a00424b59c76eaa85466" Nov 02 20:48:36.520520 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:36.520502 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da848c22a288dc7a52c88904ff7c4c9dc75a9b6b7556a00424b59c76eaa85466\": container with ID starting with da848c22a288dc7a52c88904ff7c4c9dc75a9b6b7556a00424b59c76eaa85466 not found: ID does not exist" containerID="da848c22a288dc7a52c88904ff7c4c9dc75a9b6b7556a00424b59c76eaa85466" Nov 02 20:48:36.520582 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.520532 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da848c22a288dc7a52c88904ff7c4c9dc75a9b6b7556a00424b59c76eaa85466"} err="failed to get container status \"da848c22a288dc7a52c88904ff7c4c9dc75a9b6b7556a00424b59c76eaa85466\": rpc error: code = NotFound desc = could not find container \"da848c22a288dc7a52c88904ff7c4c9dc75a9b6b7556a00424b59c76eaa85466\": container with ID starting with da848c22a288dc7a52c88904ff7c4c9dc75a9b6b7556a00424b59c76eaa85466 not found: ID does not exist" Nov 02 20:48:36.655124 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.653947 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558"] Nov 02 20:48:36.684652 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.684618 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn" Nov 02 20:48:36.686799 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.686782 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6" Nov 02 20:48:36.700363 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.700342 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-2250b4-574f446fcf-k9558"] Nov 02 20:48:36.761641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.761582 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t"] Nov 02 20:48:36.767263 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.767241 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-2250b4-7bf4bc6d46-wgc9t"] Nov 02 20:48:36.787559 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.787535 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-maas-api-openshift-ca\") pod \"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1\" (UID: \"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1\") " Nov 02 20:48:36.787650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.787606 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc54304f-438e-4dd9-8338-16b67bab73b7-kube-api-access-nbtdn\" (UniqueName: \"kubernetes.io/projected/dc54304f-438e-4dd9-8338-16b67bab73b7-kube-api-access-nbtdn\") pod \"dc54304f-438e-4dd9-8338-16b67bab73b7\" (UID: \"dc54304f-438e-4dd9-8338-16b67bab73b7\") " Nov 02 20:48:36.787650 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.787628 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc54304f-438e-4dd9-8338-16b67bab73b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc54304f-438e-4dd9-8338-16b67bab73b7-maas-api-xks-ca\") pod \"dc54304f-438e-4dd9-8338-16b67bab73b7\" (UID: \"dc54304f-438e-4dd9-8338-16b67bab73b7\") " Nov 02 20:48:36.787773 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.787653 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc54304f-438e-4dd9-8338-16b67bab73b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc54304f-438e-4dd9-8338-16b67bab73b7-maas-api-openshift-ca\") pod \"dc54304f-438e-4dd9-8338-16b67bab73b7\" (UID: \"dc54304f-438e-4dd9-8338-16b67bab73b7\") " Nov 02 20:48:36.787773 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.787673 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-maas-api-xks-ca\") pod \"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1\" (UID: \"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1\") " Nov 02 20:48:36.787773 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.787695 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-kube-api-access-9n667\" (UniqueName: \"kubernetes.io/projected/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-kube-api-access-9n667\") pod \"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1\" (UID: \"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1\") " Nov 02 20:48:36.787997 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.787974 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-maas-api-openshift-ca" pod "9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1" (UID: "9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:36.788139 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.788071 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc54304f-438e-4dd9-8338-16b67bab73b7-maas-api-openshift-ca" pod "dc54304f-438e-4dd9-8338-16b67bab73b7" (UID: "dc54304f-438e-4dd9-8338-16b67bab73b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:36.788328 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.788290 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:36.788328 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.788321 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc54304f-438e-4dd9-8338-16b67bab73b7-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:36.790070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.790039 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc54304f-438e-4dd9-8338-16b67bab73b7-maas-api-xks-ca" pod "dc54304f-438e-4dd9-8338-16b67bab73b7" (UID: "dc54304f-438e-4dd9-8338-16b67bab73b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:36.790223 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.790072 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-kube-api-access-9n667" pod "9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1" (UID: "9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1"). InnerVolumeSpecName "kube-api-access-9n667". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:36.790512 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.790482 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-maas-api-xks-ca" pod "9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1" (UID: "9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:36.790591 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.790493 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc54304f-438e-4dd9-8338-16b67bab73b7-kube-api-access-nbtdn" pod "dc54304f-438e-4dd9-8338-16b67bab73b7" (UID: "dc54304f-438e-4dd9-8338-16b67bab73b7"). InnerVolumeSpecName "kube-api-access-nbtdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:36.889980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.889954 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbtdn\" (UniqueName: \"kubernetes.io/projected/dc54304f-438e-4dd9-8338-16b67bab73b7-kube-api-access-nbtdn\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:36.890075 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.889988 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc54304f-438e-4dd9-8338-16b67bab73b7-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:36.890075 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.890004 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:36.890075 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:36.890018 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n667\" (UniqueName: \"kubernetes.io/projected/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1-kube-api-access-9n667\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:37.505205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:37.505121 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn" Nov 02 20:48:37.505205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:37.505130 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn" event={"ID":"dc54304f-438e-4dd9-8338-16b67bab73b7","Type":"ContainerDied","Data":"41b300100ee09af59927b69df5802f545c0576918931d79b86e56752246dc7b6"} Nov 02 20:48:37.505205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:37.505176 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41b300100ee09af59927b69df5802f545c0576918931d79b86e56752246dc7b6" Nov 02 20:48:37.507709 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:37.507689 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6" Nov 02 20:48:37.507866 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:37.507688 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6" event={"ID":"9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1","Type":"ContainerDied","Data":"acc13436068dc66dc14a17ae31a20c1a4aba91417ba65901e3f706b508d4205e"} Nov 02 20:48:37.507866 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:37.507807 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acc13436068dc66dc14a17ae31a20c1a4aba91417ba65901e3f706b508d4205e" Nov 02 20:48:37.865706 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:37.865670 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29a43c7-6192-4151-9c21-bcedab45ba8c" path="/var/lib/kubelet/pods/a29a43c7-6192-4151-9c21-bcedab45ba8c/volumes" Nov 02 20:48:37.866152 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:37.866132 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48" path="/var/lib/kubelet/pods/ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48/volumes" Nov 02 20:48:39.425572 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.425495 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m"] Nov 02 20:48:39.426137 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.425683 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" podUID="181451b7-ca86-4aa2-ae78-73f17b5a67d2" containerName="discovery" containerID="cri-o://8a77e8df154012a822ad3e97f8e1dabda47497f25cc95dadd4a3c4df43bee371" gracePeriod=30 Nov 02 20:48:39.604192 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.604117 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:39.707317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.707294 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:48:39.819418 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.819389 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181451b7-ca86-4aa2-ae78-73f17b5a67d2-kube-api-access-rbspp\" (UniqueName: \"kubernetes.io/projected/181451b7-ca86-4aa2-ae78-73f17b5a67d2-kube-api-access-rbspp\") pod \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " Nov 02 20:48:39.819556 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.819457 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-cacerts\" (UniqueName: \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-cacerts\") pod \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " Nov 02 20:48:39.819613 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.819579 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181451b7-ca86-4aa2-ae78-73f17b5a67d2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/181451b7-ca86-4aa2-ae78-73f17b5a67d2-local-certs\") pod \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " Nov 02 20:48:39.819653 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.819641 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-token\" (UniqueName: \"kubernetes.io/projected/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-token\") pod \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " Nov 02 20:48:39.819698 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.819683 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-kubeconfig\") pod \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " Nov 02 20:48:39.819785 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.819768 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-csr-dns-cert\") pod \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " Nov 02 20:48:39.819853 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.819802 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-csr-ca-configmap\") pod \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\" (UID: \"181451b7-ca86-4aa2-ae78-73f17b5a67d2\") " Nov 02 20:48:39.821445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.821407 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-csr-ca-configmap" pod "181451b7-ca86-4aa2-ae78-73f17b5a67d2" (UID: "181451b7-ca86-4aa2-ae78-73f17b5a67d2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:39.822499 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.822468 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-cacerts" pod "181451b7-ca86-4aa2-ae78-73f17b5a67d2" (UID: "181451b7-ca86-4aa2-ae78-73f17b5a67d2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:39.822643 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.822618 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181451b7-ca86-4aa2-ae78-73f17b5a67d2-kube-api-access-rbspp" pod "181451b7-ca86-4aa2-ae78-73f17b5a67d2" (UID: "181451b7-ca86-4aa2-ae78-73f17b5a67d2"). InnerVolumeSpecName "kube-api-access-rbspp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:39.822782 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.822665 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-token" pod "181451b7-ca86-4aa2-ae78-73f17b5a67d2" (UID: "181451b7-ca86-4aa2-ae78-73f17b5a67d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:39.822782 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.822701 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181451b7-ca86-4aa2-ae78-73f17b5a67d2-local-certs" pod "181451b7-ca86-4aa2-ae78-73f17b5a67d2" (UID: "181451b7-ca86-4aa2-ae78-73f17b5a67d2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:39.823054 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.823028 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-kubeconfig" pod "181451b7-ca86-4aa2-ae78-73f17b5a67d2" (UID: "181451b7-ca86-4aa2-ae78-73f17b5a67d2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:39.823478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.823449 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-csr-dns-cert" pod "181451b7-ca86-4aa2-ae78-73f17b5a67d2" (UID: "181451b7-ca86-4aa2-ae78-73f17b5a67d2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:39.921026 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.920998 4971 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-cacerts\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:39.921026 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.921023 4971 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/181451b7-ca86-4aa2-ae78-73f17b5a67d2-local-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:39.921279 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.921032 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:39.921279 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.921041 4971 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-kubeconfig\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:39.921279 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.921051 4971 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-csr-dns-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:39.921279 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.921060 4971 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/181451b7-ca86-4aa2-ae78-73f17b5a67d2-istio-csr-ca-configmap\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:39.921279 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:39.921069 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbspp\" (UniqueName: \"kubernetes.io/projected/181451b7-ca86-4aa2-ae78-73f17b5a67d2-kube-api-access-rbspp\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:40.529704 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:40.529671 4971 generic.go:356] "Generic (PLEG): container finished" podID="181451b7-ca86-4aa2-ae78-73f17b5a67d2" containerID="8a77e8df154012a822ad3e97f8e1dabda47497f25cc95dadd4a3c4df43bee371" exitCode=0 Nov 02 20:48:40.530118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:40.529716 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" Nov 02 20:48:40.530118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:40.529751 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" event={"ID":"181451b7-ca86-4aa2-ae78-73f17b5a67d2","Type":"ContainerDied","Data":"8a77e8df154012a822ad3e97f8e1dabda47497f25cc95dadd4a3c4df43bee371"} Nov 02 20:48:40.530118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:40.529804 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m" event={"ID":"181451b7-ca86-4aa2-ae78-73f17b5a67d2","Type":"ContainerDied","Data":"709e52eff676a060490332bb9dc7f5940d0b0b729bc502dfb1d5895adb57fc9f"} Nov 02 20:48:40.530118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:40.529832 4971 scope.go:122] "RemoveContainer" containerID="8a77e8df154012a822ad3e97f8e1dabda47497f25cc95dadd4a3c4df43bee371" Nov 02 20:48:40.540449 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:40.540433 4971 scope.go:122] "RemoveContainer" containerID="8a77e8df154012a822ad3e97f8e1dabda47497f25cc95dadd4a3c4df43bee371" Nov 02 20:48:40.540726 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:40.540703 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a77e8df154012a822ad3e97f8e1dabda47497f25cc95dadd4a3c4df43bee371\": container with ID starting with 8a77e8df154012a822ad3e97f8e1dabda47497f25cc95dadd4a3c4df43bee371 not found: ID does not exist" containerID="8a77e8df154012a822ad3e97f8e1dabda47497f25cc95dadd4a3c4df43bee371" Nov 02 20:48:40.540793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:40.540739 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a77e8df154012a822ad3e97f8e1dabda47497f25cc95dadd4a3c4df43bee371"} err="failed to get container status \"8a77e8df154012a822ad3e97f8e1dabda47497f25cc95dadd4a3c4df43bee371\": rpc error: code = NotFound desc = could not find container \"8a77e8df154012a822ad3e97f8e1dabda47497f25cc95dadd4a3c4df43bee371\": container with ID starting with 8a77e8df154012a822ad3e97f8e1dabda47497f25cc95dadd4a3c4df43bee371 not found: ID does not exist" Nov 02 20:48:40.691239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:40.691215 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx"] Nov 02 20:48:40.691438 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:40.691415 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" podUID="0d6924ca-50c1-496f-9bd0-fcaa96717981" containerName="istio-proxy" containerID="cri-o://326a896a86532092e8b8e75ccc7b68d1b17bdc95e320c2911a9b8a99b1c96d3d" gracePeriod=30 Nov 02 20:48:40.732981 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:40.732952 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m"] Nov 02 20:48:40.737728 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:40.737707 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sl97m"] Nov 02 20:48:41.260262 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:41.260194 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:41.517409 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:41.517292 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:41.770395 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:41.770310 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:41.856692 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:41.856660 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:48:41.856836 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:41.856716 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" podUID="0d6924ca-50c1-496f-9bd0-fcaa96717981" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:48:41.866369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:41.866345 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181451b7-ca86-4aa2-ae78-73f17b5a67d2" path="/var/lib/kubelet/pods/181451b7-ca86-4aa2-ae78-73f17b5a67d2/volumes" Nov 02 20:48:42.017123 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:42.013598 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:42.109065 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:42.109025 4971 scope.go:122] "RemoveContainer" containerID="6bd25a762c899fc3fb423ba4a3caa2cfb1e6af565796b2c41cde4e91dd2707b1" Nov 02 20:48:42.121580 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:42.121552 4971 scope.go:122] "RemoveContainer" containerID="e81064b4a36e66691fcdcd2d84869716ee9139d8944442e20fc125005249b468" Nov 02 20:48:42.413851 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:42.413773 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:43.039340 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.039307 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk"] Nov 02 20:48:43.039806 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.039793 4971 cpu_manager.go:414] "containerMap: removing container" podUID="ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48" containerName="payload-pre-processing" Nov 02 20:48:43.039868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.039807 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48" containerName="payload-pre-processing" Nov 02 20:48:43.039868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.039819 4971 cpu_manager.go:414] "containerMap: removing container" podUID="a29a43c7-6192-4151-9c21-bcedab45ba8c" containerName="payload-processing" Nov 02 20:48:43.039868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.039825 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29a43c7-6192-4151-9c21-bcedab45ba8c" containerName="payload-processing" Nov 02 20:48:43.039868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.039844 4971 cpu_manager.go:414] "containerMap: removing container" podUID="181451b7-ca86-4aa2-ae78-73f17b5a67d2" containerName="discovery" Nov 02 20:48:43.039868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.039850 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181451b7-ca86-4aa2-ae78-73f17b5a67d2" containerName="discovery" Nov 02 20:48:43.039868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.039864 4971 cpu_manager.go:414] "containerMap: removing container" podUID="9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1" containerName="revoke-keys" Nov 02 20:48:43.039868 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.039868 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1" containerName="revoke-keys" Nov 02 20:48:43.040125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.039885 4971 cpu_manager.go:414] "containerMap: removing container" podUID="dc54304f-438e-4dd9-8338-16b67bab73b7" containerName="revoke-keys" Nov 02 20:48:43.040125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.039891 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc54304f-438e-4dd9-8338-16b67bab73b7" containerName="revoke-keys" Nov 02 20:48:43.040125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.039963 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc54304f-438e-4dd9-8338-16b67bab73b7" containerName="revoke-keys" Nov 02 20:48:43.040125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.039974 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad50ecfc-027a-4ec8-adf7-f3bd20ae4c48" containerName="payload-pre-processing" Nov 02 20:48:43.040125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.039981 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29a43c7-6192-4151-9c21-bcedab45ba8c" containerName="payload-processing" Nov 02 20:48:43.040125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.039990 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="181451b7-ca86-4aa2-ae78-73f17b5a67d2" containerName="discovery" Nov 02 20:48:43.040125 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.039997 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1" containerName="revoke-keys" Nov 02 20:48:43.045499 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.045478 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" Nov 02 20:48:43.096503 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.096474 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:48:43.100253 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.100223 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk"] Nov 02 20:48:43.142801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.142771 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:48:43.153879 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.153849 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045025e5-eabc-43de-bc13-681d499ceced-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk\" (UID: \"045025e5-eabc-43de-bc13-681d499ceced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" Nov 02 20:48:43.154071 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.153934 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045025e5-eabc-43de-bc13-681d499ceced-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk\" (UID: \"045025e5-eabc-43de-bc13-681d499ceced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" Nov 02 20:48:43.154200 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.154124 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wknh9\" (UniqueName: \"kubernetes.io/projected/045025e5-eabc-43de-bc13-681d499ceced-kube-api-access-wknh9\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk\" (UID: \"045025e5-eabc-43de-bc13-681d499ceced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" Nov 02 20:48:43.255163 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.255124 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045025e5-eabc-43de-bc13-681d499ceced-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk\" (UID: \"045025e5-eabc-43de-bc13-681d499ceced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" Nov 02 20:48:43.255322 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.255221 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045025e5-eabc-43de-bc13-681d499ceced-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk\" (UID: \"045025e5-eabc-43de-bc13-681d499ceced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" Nov 02 20:48:43.255369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.255349 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wknh9\" (UniqueName: \"kubernetes.io/projected/045025e5-eabc-43de-bc13-681d499ceced-kube-api-access-wknh9\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk\" (UID: \"045025e5-eabc-43de-bc13-681d499ceced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" Nov 02 20:48:43.255926 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.255903 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045025e5-eabc-43de-bc13-681d499ceced-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk\" (UID: \"045025e5-eabc-43de-bc13-681d499ceced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" Nov 02 20:48:43.257738 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.257717 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045025e5-eabc-43de-bc13-681d499ceced-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk\" (UID: \"045025e5-eabc-43de-bc13-681d499ceced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" Nov 02 20:48:43.313451 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.313385 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wknh9\" (UniqueName: \"kubernetes.io/projected/045025e5-eabc-43de-bc13-681d499ceced-kube-api-access-wknh9\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk\" (UID: \"045025e5-eabc-43de-bc13-681d499ceced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" Nov 02 20:48:43.359294 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.359264 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" Nov 02 20:48:43.730826 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:43.730556 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk"] Nov 02 20:48:43.731137 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:48:43.731055 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045025e5_eabc_43de_bc13_681d499ceced.slice/crio-6dcbb6579cc2e23e7808046f99d0a3abbc5fbac3d7d9ff36af1ec40521ad2f7a WatchSource:0}: Error finding container 6dcbb6579cc2e23e7808046f99d0a3abbc5fbac3d7d9ff36af1ec40521ad2f7a: Status 404 returned error can't find the container with id 6dcbb6579cc2e23e7808046f99d0a3abbc5fbac3d7d9ff36af1ec40521ad2f7a Nov 02 20:48:44.302348 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:44.302228 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:44.561781 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:44.561693 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" event={"ID":"045025e5-eabc-43de-bc13-681d499ceced","Type":"ContainerStarted","Data":"533843b7951384c2629b8d6f55e6f7edd9ecc717f6a5e58f4ecde2142b1d861a"} Nov 02 20:48:44.561971 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:44.561795 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" event={"ID":"045025e5-eabc-43de-bc13-681d499ceced","Type":"ContainerStarted","Data":"6dcbb6579cc2e23e7808046f99d0a3abbc5fbac3d7d9ff36af1ec40521ad2f7a"} Nov 02 20:48:44.675283 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:44.675217 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" podStartSLOduration=2.675197317 podStartE2EDuration="2.675197317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:48:44.671630549 +0000 UTC m=+2052.961287102" watchObservedRunningTime="2026-11-02 20:48:44.675197317 +0000 UTC m=+2052.964853849" Nov 02 20:48:45.449388 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:45.449352 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:45.573870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:45.573837 4971 generic.go:356] "Generic (PLEG): container finished" podID="045025e5-eabc-43de-bc13-681d499ceced" containerID="533843b7951384c2629b8d6f55e6f7edd9ecc717f6a5e58f4ecde2142b1d861a" exitCode=0 Nov 02 20:48:45.574056 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:45.573875 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" event={"ID":"045025e5-eabc-43de-bc13-681d499ceced","Type":"ContainerDied","Data":"533843b7951384c2629b8d6f55e6f7edd9ecc717f6a5e58f4ecde2142b1d861a"} Nov 02 20:48:45.899867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:45.899834 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:45.971039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:45.970977 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:48:46.093946 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.093915 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-token\" (UniqueName: \"kubernetes.io/projected/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-token\") pod \"0d6924ca-50c1-496f-9bd0-fcaa96717981\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " Nov 02 20:48:46.094177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.093960 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-credential-socket\") pod \"0d6924ca-50c1-496f-9bd0-fcaa96717981\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " Nov 02 20:48:46.094177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.093988 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-envoy\") pod \"0d6924ca-50c1-496f-9bd0-fcaa96717981\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " Nov 02 20:48:46.094177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.094019 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6924ca-50c1-496f-9bd0-fcaa96717981-kube-api-access-flr6f\" (UniqueName: \"kubernetes.io/projected/0d6924ca-50c1-496f-9bd0-fcaa96717981-kube-api-access-flr6f\") pod \"0d6924ca-50c1-496f-9bd0-fcaa96717981\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " Nov 02 20:48:46.094177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.094120 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-workload-socket\") pod \"0d6924ca-50c1-496f-9bd0-fcaa96717981\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " Nov 02 20:48:46.094445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.094251 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-podinfo\") pod \"0d6924ca-50c1-496f-9bd0-fcaa96717981\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " Nov 02 20:48:46.094445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.094320 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-workload-certs\") pod \"0d6924ca-50c1-496f-9bd0-fcaa96717981\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " Nov 02 20:48:46.094445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.094366 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d6924ca-50c1-496f-9bd0-fcaa96717981-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6924ca-50c1-496f-9bd0-fcaa96717981-istiod-ca-cert\") pod \"0d6924ca-50c1-496f-9bd0-fcaa96717981\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " Nov 02 20:48:46.094445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.094420 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-data\") pod \"0d6924ca-50c1-496f-9bd0-fcaa96717981\" (UID: \"0d6924ca-50c1-496f-9bd0-fcaa96717981\") " Nov 02 20:48:46.094645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.094621 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-credential-socket" pod "0d6924ca-50c1-496f-9bd0-fcaa96717981" (UID: "0d6924ca-50c1-496f-9bd0-fcaa96717981"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:46.094939 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.094832 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-data" pod "0d6924ca-50c1-496f-9bd0-fcaa96717981" (UID: "0d6924ca-50c1-496f-9bd0-fcaa96717981"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:46.095109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.094995 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:46.095109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.095021 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:46.095206 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.095071 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-workload-certs" pod "0d6924ca-50c1-496f-9bd0-fcaa96717981" (UID: "0d6924ca-50c1-496f-9bd0-fcaa96717981"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:46.095306 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.095280 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6924ca-50c1-496f-9bd0-fcaa96717981-istiod-ca-cert" pod "0d6924ca-50c1-496f-9bd0-fcaa96717981" (UID: "0d6924ca-50c1-496f-9bd0-fcaa96717981"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:46.095375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.095357 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-workload-socket" pod "0d6924ca-50c1-496f-9bd0-fcaa96717981" (UID: "0d6924ca-50c1-496f-9bd0-fcaa96717981"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:46.097018 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.096986 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-envoy" pod "0d6924ca-50c1-496f-9bd0-fcaa96717981" (UID: "0d6924ca-50c1-496f-9bd0-fcaa96717981"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:46.097526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.097487 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6924ca-50c1-496f-9bd0-fcaa96717981-kube-api-access-flr6f" pod "0d6924ca-50c1-496f-9bd0-fcaa96717981" (UID: "0d6924ca-50c1-496f-9bd0-fcaa96717981"). InnerVolumeSpecName "kube-api-access-flr6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:46.097768 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.097743 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-token" pod "0d6924ca-50c1-496f-9bd0-fcaa96717981" (UID: "0d6924ca-50c1-496f-9bd0-fcaa96717981"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:46.098224 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.098196 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-podinfo" pod "0d6924ca-50c1-496f-9bd0-fcaa96717981" (UID: "0d6924ca-50c1-496f-9bd0-fcaa96717981"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:48:46.187946 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.187911 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:46.196341 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.196306 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:46.196341 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.196337 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:46.196526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.196352 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6924ca-50c1-496f-9bd0-fcaa96717981-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:46.196526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.196365 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:46.196526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.196377 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:46.196526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.196391 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flr6f\" (UniqueName: \"kubernetes.io/projected/0d6924ca-50c1-496f-9bd0-fcaa96717981-kube-api-access-flr6f\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:46.196526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.196404 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6924ca-50c1-496f-9bd0-fcaa96717981-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:46.344400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.344282 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc"] Nov 02 20:48:46.347892 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.347867 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-64ff15-hqzhc"] Nov 02 20:48:46.430974 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.430919 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:46.582116 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.582061 4971 generic.go:356] "Generic (PLEG): container finished" podID="0d6924ca-50c1-496f-9bd0-fcaa96717981" containerID="326a896a86532092e8b8e75ccc7b68d1b17bdc95e320c2911a9b8a99b1c96d3d" exitCode=0 Nov 02 20:48:46.582567 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.582153 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" Nov 02 20:48:46.582567 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.582153 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" event={"ID":"0d6924ca-50c1-496f-9bd0-fcaa96717981","Type":"ContainerDied","Data":"326a896a86532092e8b8e75ccc7b68d1b17bdc95e320c2911a9b8a99b1c96d3d"} Nov 02 20:48:46.582567 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.582238 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx" event={"ID":"0d6924ca-50c1-496f-9bd0-fcaa96717981","Type":"ContainerDied","Data":"ee8002fb4f56766ac94fe8745f4a94e95805be60ab1b9c588695f44a0259a8e6"} Nov 02 20:48:46.582567 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.582277 4971 scope.go:122] "RemoveContainer" containerID="326a896a86532092e8b8e75ccc7b68d1b17bdc95e320c2911a9b8a99b1c96d3d" Nov 02 20:48:46.594832 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.594813 4971 scope.go:122] "RemoveContainer" containerID="326a896a86532092e8b8e75ccc7b68d1b17bdc95e320c2911a9b8a99b1c96d3d" Nov 02 20:48:46.595151 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:46.595130 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"326a896a86532092e8b8e75ccc7b68d1b17bdc95e320c2911a9b8a99b1c96d3d\": container with ID starting with 326a896a86532092e8b8e75ccc7b68d1b17bdc95e320c2911a9b8a99b1c96d3d not found: ID does not exist" containerID="326a896a86532092e8b8e75ccc7b68d1b17bdc95e320c2911a9b8a99b1c96d3d" Nov 02 20:48:46.595218 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.595160 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"326a896a86532092e8b8e75ccc7b68d1b17bdc95e320c2911a9b8a99b1c96d3d"} err="failed to get container status \"326a896a86532092e8b8e75ccc7b68d1b17bdc95e320c2911a9b8a99b1c96d3d\": rpc error: code = NotFound desc = could not find container \"326a896a86532092e8b8e75ccc7b68d1b17bdc95e320c2911a9b8a99b1c96d3d\": container with ID starting with 326a896a86532092e8b8e75ccc7b68d1b17bdc95e320c2911a9b8a99b1c96d3d not found: ID does not exist" Nov 02 20:48:46.707368 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.707344 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" Nov 02 20:48:46.738134 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.738108 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx"] Nov 02 20:48:46.784978 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.784950 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-2bdkx"] Nov 02 20:48:46.802252 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.802223 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045025e5-eabc-43de-bc13-681d499ceced-kube-api-access-wknh9\" (UniqueName: \"kubernetes.io/projected/045025e5-eabc-43de-bc13-681d499ceced-kube-api-access-wknh9\") pod \"045025e5-eabc-43de-bc13-681d499ceced\" (UID: \"045025e5-eabc-43de-bc13-681d499ceced\") " Nov 02 20:48:46.802373 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.802338 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/045025e5-eabc-43de-bc13-681d499ceced-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045025e5-eabc-43de-bc13-681d499ceced-maas-api-openshift-ca\") pod \"045025e5-eabc-43de-bc13-681d499ceced\" (UID: \"045025e5-eabc-43de-bc13-681d499ceced\") " Nov 02 20:48:46.802440 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.802392 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/045025e5-eabc-43de-bc13-681d499ceced-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045025e5-eabc-43de-bc13-681d499ceced-maas-api-xks-ca\") pod \"045025e5-eabc-43de-bc13-681d499ceced\" (UID: \"045025e5-eabc-43de-bc13-681d499ceced\") " Nov 02 20:48:46.802668 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.802647 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045025e5-eabc-43de-bc13-681d499ceced-maas-api-openshift-ca" pod "045025e5-eabc-43de-bc13-681d499ceced" (UID: "045025e5-eabc-43de-bc13-681d499ceced"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:46.802795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.802778 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/045025e5-eabc-43de-bc13-681d499ceced-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:46.804537 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.804511 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/045025e5-eabc-43de-bc13-681d499ceced-maas-api-xks-ca" pod "045025e5-eabc-43de-bc13-681d499ceced" (UID: "045025e5-eabc-43de-bc13-681d499ceced"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:46.804642 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.804589 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045025e5-eabc-43de-bc13-681d499ceced-kube-api-access-wknh9" pod "045025e5-eabc-43de-bc13-681d499ceced" (UID: "045025e5-eabc-43de-bc13-681d499ceced"). InnerVolumeSpecName "kube-api-access-wknh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:46.879394 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.879365 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:46.904301 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.904276 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wknh9\" (UniqueName: \"kubernetes.io/projected/045025e5-eabc-43de-bc13-681d499ceced-kube-api-access-wknh9\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:46.904301 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:46.904298 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/045025e5-eabc-43de-bc13-681d499ceced-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:47.591744 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:47.591719 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" Nov 02 20:48:47.592177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:47.591791 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk" event={"ID":"045025e5-eabc-43de-bc13-681d499ceced","Type":"ContainerDied","Data":"6dcbb6579cc2e23e7808046f99d0a3abbc5fbac3d7d9ff36af1ec40521ad2f7a"} Nov 02 20:48:47.592177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:47.591834 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dcbb6579cc2e23e7808046f99d0a3abbc5fbac3d7d9ff36af1ec40521ad2f7a" Nov 02 20:48:47.866238 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:47.866207 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6924ca-50c1-496f-9bd0-fcaa96717981" path="/var/lib/kubelet/pods/0d6924ca-50c1-496f-9bd0-fcaa96717981/volumes" Nov 02 20:48:47.866635 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:47.866620 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="985b395c-deed-4e79-b13e-5d8566f29218" path="/var/lib/kubelet/pods/985b395c-deed-4e79-b13e-5d8566f29218/volumes" Nov 02 20:48:48.592788 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:48.592747 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:49.046913 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:49.046877 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh"] Nov 02 20:48:49.093780 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:49.093749 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a4600b-7kmmh"] Nov 02 20:48:49.866390 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:49.866360 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c278625-8b76-4f43-95a3-510cb72ea044" path="/var/lib/kubelet/pods/1c278625-8b76-4f43-95a3-510cb72ea044/volumes" Nov 02 20:48:50.045523 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:50.045490 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m"] Nov 02 20:48:50.092002 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:50.091974 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-893a0a2c-rhs7m"] Nov 02 20:48:50.921227 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:50.921192 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:51.865921 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:51.865884 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c0b5b7d-6e10-4dea-80df-69f60f219ff7" path="/var/lib/kubelet/pods/3c0b5b7d-6e10-4dea-80df-69f60f219ff7/volumes" Nov 02 20:48:52.565592 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:52.565550 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:52.830252 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:52.830184 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:52.945505 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:52.945469 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9"] Nov 02 20:48:52.992695 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:52.992659 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-30ee89-9p9h9"] Nov 02 20:48:53.076454 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:53.076429 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:53.105030 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:53.104999 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p"] Nov 02 20:48:53.105290 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:53.105263 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" podUID="1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" containerName="istio-proxy" containerID="cri-o://1b6c26b1e6aabc1867eeedad8e1db9ce55fee69121f098d49673565952a48cbc" gracePeriod=30 Nov 02 20:48:53.326369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:53.326323 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:53.574075 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:53.573975 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:53.818110 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:53.818052 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:53.872610 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:53.872578 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9063aea-4796-49d4-b7c3-e7ae34292bc8" path="/var/lib/kubelet/pods/c9063aea-4796-49d4-b7c3-e7ae34292bc8/volumes" Nov 02 20:48:54.081648 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.081616 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr"] Nov 02 20:48:54.082099 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.082071 4971 cpu_manager.go:414] "containerMap: removing container" podUID="0d6924ca-50c1-496f-9bd0-fcaa96717981" containerName="istio-proxy" Nov 02 20:48:54.082159 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.082099 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6924ca-50c1-496f-9bd0-fcaa96717981" containerName="istio-proxy" Nov 02 20:48:54.082159 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.082125 4971 cpu_manager.go:414] "containerMap: removing container" podUID="045025e5-eabc-43de-bc13-681d499ceced" containerName="revoke-keys" Nov 02 20:48:54.082159 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.082132 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045025e5-eabc-43de-bc13-681d499ceced" containerName="revoke-keys" Nov 02 20:48:54.082270 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.082229 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="045025e5-eabc-43de-bc13-681d499ceced" containerName="revoke-keys" Nov 02 20:48:54.082270 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.082247 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6924ca-50c1-496f-9bd0-fcaa96717981" containerName="istio-proxy" Nov 02 20:48:54.085638 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.085612 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr" Nov 02 20:48:54.135004 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.134953 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:48:54.135004 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.134982 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:48:54.170466 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.170439 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqg2l\" (UniqueName: \"kubernetes.io/projected/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-kube-api-access-sqg2l\") pod \"maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr\" (UID: \"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr" Nov 02 20:48:54.170601 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.170494 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr\" (UID: \"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr" Nov 02 20:48:54.170765 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.170739 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr\" (UID: \"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr" Nov 02 20:48:54.184570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.184533 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr"] Nov 02 20:48:54.271437 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.271410 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr\" (UID: \"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr" Nov 02 20:48:54.271552 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.271461 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqg2l\" (UniqueName: \"kubernetes.io/projected/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-kube-api-access-sqg2l\") pod \"maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr\" (UID: \"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr" Nov 02 20:48:54.271552 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.271492 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr\" (UID: \"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr" Nov 02 20:48:54.272122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.272102 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr\" (UID: \"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr" Nov 02 20:48:54.273906 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.273885 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr\" (UID: \"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr" Nov 02 20:48:54.329348 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.329322 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqg2l\" (UniqueName: \"kubernetes.io/projected/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-kube-api-access-sqg2l\") pod \"maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr\" (UID: \"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr" Nov 02 20:48:54.399371 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.399311 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr" Nov 02 20:48:54.768406 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.768366 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr"] Nov 02 20:48:54.769145 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:48:54.769111 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c85642_5d6f_49de_8ba4_5cd2a9f543d0.slice/crio-0ddf6e3bd5fa84220ac3dfdecc6a5cf09308216a71f1b6a5d69a087aa4bdee10 WatchSource:0}: Error finding container 0ddf6e3bd5fa84220ac3dfdecc6a5cf09308216a71f1b6a5d69a087aa4bdee10: Status 404 returned error can't find the container with id 0ddf6e3bd5fa84220ac3dfdecc6a5cf09308216a71f1b6a5d69a087aa4bdee10 Nov 02 20:48:54.877631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.877596 4971 patch_prober.go:29] interesting pod/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:48:54.877759 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:54.877653 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" podUID="1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:48:55.644052 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:55.644023 4971 generic.go:356] "Generic (PLEG): container finished" podID="c7c85642-5d6f-49de-8ba4-5cd2a9f543d0" containerID="71227b01d4e57e7cf3a521e773269ec6bb7249219b81880f7a00c468de91876e" exitCode=0 Nov 02 20:48:55.644214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:55.644115 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr" event={"ID":"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0","Type":"ContainerDied","Data":"71227b01d4e57e7cf3a521e773269ec6bb7249219b81880f7a00c468de91876e"} Nov 02 20:48:55.644214 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:55.644171 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr" event={"ID":"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0","Type":"ContainerStarted","Data":"0ddf6e3bd5fa84220ac3dfdecc6a5cf09308216a71f1b6a5d69a087aa4bdee10"} Nov 02 20:48:55.812912 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:55.812885 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn"] Nov 02 20:48:55.923019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:55.922957 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs"] Nov 02 20:48:56.289433 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:56.289349 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:56.544994 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:56.544915 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:56.803334 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:56.803281 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr" Nov 02 20:48:56.895154 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:56.895119 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-maas-api-xks-ca\") pod \"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0\" (UID: \"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0\") " Nov 02 20:48:56.895583 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:56.895238 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-kube-api-access-sqg2l\" (UniqueName: \"kubernetes.io/projected/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-kube-api-access-sqg2l\") pod \"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0\" (UID: \"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0\") " Nov 02 20:48:56.895583 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:56.895285 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-maas-api-openshift-ca\") pod \"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0\" (UID: \"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0\") " Nov 02 20:48:56.895726 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:56.895686 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-maas-api-openshift-ca" pod "c7c85642-5d6f-49de-8ba4-5cd2a9f543d0" (UID: "c7c85642-5d6f-49de-8ba4-5cd2a9f543d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:56.897360 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:56.897334 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-maas-api-xks-ca" pod "c7c85642-5d6f-49de-8ba4-5cd2a9f543d0" (UID: "c7c85642-5d6f-49de-8ba4-5cd2a9f543d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:56.897500 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:56.897481 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-kube-api-access-sqg2l" pod "c7c85642-5d6f-49de-8ba4-5cd2a9f543d0" (UID: "c7c85642-5d6f-49de-8ba4-5cd2a9f543d0"). InnerVolumeSpecName "kube-api-access-sqg2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:56.997226 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:56.997195 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:56.997226 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:56.997229 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqg2l\" (UniqueName: \"kubernetes.io/projected/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-kube-api-access-sqg2l\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:56.997449 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:56.997239 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:57.420519 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:57.420482 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:57.662544 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:57.662509 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr" Nov 02 20:48:57.662735 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:57.662546 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr" event={"ID":"c7c85642-5d6f-49de-8ba4-5cd2a9f543d0","Type":"ContainerDied","Data":"0ddf6e3bd5fa84220ac3dfdecc6a5cf09308216a71f1b6a5d69a087aa4bdee10"} Nov 02 20:48:57.662735 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:57.662614 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ddf6e3bd5fa84220ac3dfdecc6a5cf09308216a71f1b6a5d69a087aa4bdee10" Nov 02 20:48:58.385680 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.385616 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:48:58.415351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.414339 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-envoy\") pod \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " Nov 02 20:48:58.415351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.414455 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-kube-api-access-9nxg4\" (UniqueName: \"kubernetes.io/projected/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-kube-api-access-9nxg4\") pod \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " Nov 02 20:48:58.415351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.414491 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-podinfo\") pod \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " Nov 02 20:48:58.415351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.414539 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-credential-socket\") pod \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " Nov 02 20:48:58.415351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.414650 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istiod-ca-cert\") pod \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " Nov 02 20:48:58.415351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.414690 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-data\") pod \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " Nov 02 20:48:58.415351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.414726 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-token\" (UniqueName: \"kubernetes.io/projected/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-token\") pod \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " Nov 02 20:48:58.415351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.414767 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-workload-certs\") pod \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " Nov 02 20:48:58.415351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.414793 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-workload-socket\") pod \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\" (UID: \"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3\") " Nov 02 20:48:58.415944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.415410 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istiod-ca-cert" pod "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" (UID: "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:58.415944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.415652 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-workload-certs" pod "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" (UID: "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:58.415944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.415718 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-credential-socket" pod "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" (UID: "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:58.415944 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.415851 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-data" pod "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" (UID: "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:58.416302 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.416258 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-workload-socket" pod "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" (UID: "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:58.419200 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.419174 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-token" pod "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" (UID: "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:58.420234 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.420212 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-podinfo" pod "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" (UID: "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:48:58.421746 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.421727 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-kube-api-access-9nxg4" pod "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" (UID: "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3"). InnerVolumeSpecName "kube-api-access-9nxg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:58.423173 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.423154 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-envoy" pod "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" (UID: "1ab9301b-293b-420e-8fb7-ad6eeb6acaa3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:58.516122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.516072 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:58.516280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.516126 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nxg4\" (UniqueName: \"kubernetes.io/projected/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-kube-api-access-9nxg4\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:58.516280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.516141 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:58.516280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.516153 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:58.516280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.516165 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:58.516280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.516176 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:58.516280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.516189 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:58.516280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.516200 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:58.516280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.516212 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:58.643824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.643746 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:58.674993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.674958 4971 generic.go:356] "Generic (PLEG): container finished" podID="1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" containerID="1b6c26b1e6aabc1867eeedad8e1db9ce55fee69121f098d49673565952a48cbc" exitCode=0 Nov 02 20:48:58.675168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.675015 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" Nov 02 20:48:58.675168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.675025 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" event={"ID":"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3","Type":"ContainerDied","Data":"1b6c26b1e6aabc1867eeedad8e1db9ce55fee69121f098d49673565952a48cbc"} Nov 02 20:48:58.675168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.675125 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p" event={"ID":"1ab9301b-293b-420e-8fb7-ad6eeb6acaa3","Type":"ContainerDied","Data":"953441e0c60afe53901133ee72e81b77467bd742a73239d3cd5ad180d13b1cfa"} Nov 02 20:48:58.675168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.675158 4971 scope.go:122] "RemoveContainer" containerID="1b6c26b1e6aabc1867eeedad8e1db9ce55fee69121f098d49673565952a48cbc" Nov 02 20:48:58.686480 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.686461 4971 scope.go:122] "RemoveContainer" containerID="9833fdf1e58051ac04e9ebac069cb4f3052cb38a7762058a31623d9bc1c83c1f" Nov 02 20:48:58.705926 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.705805 4971 scope.go:122] "RemoveContainer" containerID="1b6c26b1e6aabc1867eeedad8e1db9ce55fee69121f098d49673565952a48cbc" Nov 02 20:48:58.707021 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:58.706985 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b6c26b1e6aabc1867eeedad8e1db9ce55fee69121f098d49673565952a48cbc\": container with ID starting with 1b6c26b1e6aabc1867eeedad8e1db9ce55fee69121f098d49673565952a48cbc not found: ID does not exist" containerID="1b6c26b1e6aabc1867eeedad8e1db9ce55fee69121f098d49673565952a48cbc" Nov 02 20:48:58.707186 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.707035 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6c26b1e6aabc1867eeedad8e1db9ce55fee69121f098d49673565952a48cbc"} err="failed to get container status \"1b6c26b1e6aabc1867eeedad8e1db9ce55fee69121f098d49673565952a48cbc\": rpc error: code = NotFound desc = could not find container \"1b6c26b1e6aabc1867eeedad8e1db9ce55fee69121f098d49673565952a48cbc\": container with ID starting with 1b6c26b1e6aabc1867eeedad8e1db9ce55fee69121f098d49673565952a48cbc not found: ID does not exist" Nov 02 20:48:58.707186 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.707057 4971 scope.go:122] "RemoveContainer" containerID="9833fdf1e58051ac04e9ebac069cb4f3052cb38a7762058a31623d9bc1c83c1f" Nov 02 20:48:58.707443 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:48:58.707417 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9833fdf1e58051ac04e9ebac069cb4f3052cb38a7762058a31623d9bc1c83c1f\": container with ID starting with 9833fdf1e58051ac04e9ebac069cb4f3052cb38a7762058a31623d9bc1c83c1f not found: ID does not exist" containerID="9833fdf1e58051ac04e9ebac069cb4f3052cb38a7762058a31623d9bc1c83c1f" Nov 02 20:48:58.707517 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.707453 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9833fdf1e58051ac04e9ebac069cb4f3052cb38a7762058a31623d9bc1c83c1f"} err="failed to get container status \"9833fdf1e58051ac04e9ebac069cb4f3052cb38a7762058a31623d9bc1c83c1f\": rpc error: code = NotFound desc = could not find container \"9833fdf1e58051ac04e9ebac069cb4f3052cb38a7762058a31623d9bc1c83c1f\": container with ID starting with 9833fdf1e58051ac04e9ebac069cb4f3052cb38a7762058a31623d9bc1c83c1f not found: ID does not exist" Nov 02 20:48:58.833521 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.833488 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p"] Nov 02 20:48:58.843897 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.843861 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-64ff15-openshift-default-5ffdf9545-9rd4p"] Nov 02 20:48:58.895662 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:58.895582 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:59.144666 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:59.144637 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:59.390040 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:59.390007 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:48:59.463138 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:59.463036 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k"] Nov 02 20:48:59.468317 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:59.468287 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2250b4-q277k"] Nov 02 20:48:59.866164 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:59.866134 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" path="/var/lib/kubelet/pods/1ab9301b-293b-420e-8fb7-ad6eeb6acaa3/volumes" Nov 02 20:48:59.866565 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:59.866550 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9895c4c3-2e66-47a8-9d92-8172c289204b" path="/var/lib/kubelet/pods/9895c4c3-2e66-47a8-9d92-8172c289204b/volumes" Nov 02 20:48:59.904429 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:48:59.904402 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:00.333367 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:00.333284 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:00.529077 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:00.529036 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4"] Nov 02 20:49:00.529553 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:00.529279 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" podUID="dfcede60-ff17-4d6a-b3ab-65dea8834b5e" containerName="istio-proxy" containerID="cri-o://c51eee1beefea6d7d5e4ab4838f8302c202dcce84b1f9f51fdff5b5797cb84b8" gracePeriod=30 Nov 02 20:49:00.611678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:00.611643 4971 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:49:00.611894 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:00.611721 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" podUID="dfcede60-ff17-4d6a-b3ab-65dea8834b5e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:49:00.738625 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:00.738597 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk"] Nov 02 20:49:00.741342 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:00.741317 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-58c2e0-q2wwk"] Nov 02 20:49:00.814005 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:00.813965 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" podUID="2e4e9d3b-0707-402c-b5ae-7d09880c4f18" containerName="payload-processing" containerID="cri-o://1b8681c41eda5cbd0e450e42cc6fde6fc6ee63123c51c9b7e8ccb7ddb980ab42" gracePeriod=25 Nov 02 20:49:00.916631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:00.916555 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:00.923655 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:00.923624 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" podUID="31ddaa0e-4ac9-419b-a751-4351874a0a87" containerName="payload-pre-processing" containerID="cri-o://b82eb6b49080ec54e0cf9466eea230aa90946ec8c96c843146a44c3f5509bf0b" gracePeriod=25 Nov 02 20:49:01.085728 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.085694 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" Nov 02 20:49:01.140370 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.140340 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4e9d3b-0707-402c-b5ae-7d09880c4f18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4e9d3b-0707-402c-b5ae-7d09880c4f18-plugins-config-volume\") pod \"2e4e9d3b-0707-402c-b5ae-7d09880c4f18\" (UID: \"2e4e9d3b-0707-402c-b5ae-7d09880c4f18\") " Nov 02 20:49:01.140584 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.140564 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4e9d3b-0707-402c-b5ae-7d09880c4f18-kube-api-access-2f2cv\" (UniqueName: \"kubernetes.io/projected/2e4e9d3b-0707-402c-b5ae-7d09880c4f18-kube-api-access-2f2cv\") pod \"2e4e9d3b-0707-402c-b5ae-7d09880c4f18\" (UID: \"2e4e9d3b-0707-402c-b5ae-7d09880c4f18\") " Nov 02 20:49:01.140752 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.140726 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4e9d3b-0707-402c-b5ae-7d09880c4f18-plugins-config-volume" pod "2e4e9d3b-0707-402c-b5ae-7d09880c4f18" (UID: "2e4e9d3b-0707-402c-b5ae-7d09880c4f18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:01.141001 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.140974 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4e9d3b-0707-402c-b5ae-7d09880c4f18-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:01.142919 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.142892 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4e9d3b-0707-402c-b5ae-7d09880c4f18-kube-api-access-2f2cv" pod "2e4e9d3b-0707-402c-b5ae-7d09880c4f18" (UID: "2e4e9d3b-0707-402c-b5ae-7d09880c4f18"). InnerVolumeSpecName "kube-api-access-2f2cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:01.161638 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.161614 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:01.178046 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.178027 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" Nov 02 20:49:01.241463 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.241440 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ddaa0e-4ac9-419b-a751-4351874a0a87-kube-api-access-b6mfl\" (UniqueName: \"kubernetes.io/projected/31ddaa0e-4ac9-419b-a751-4351874a0a87-kube-api-access-b6mfl\") pod \"31ddaa0e-4ac9-419b-a751-4351874a0a87\" (UID: \"31ddaa0e-4ac9-419b-a751-4351874a0a87\") " Nov 02 20:49:01.241570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.241510 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ddaa0e-4ac9-419b-a751-4351874a0a87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ddaa0e-4ac9-419b-a751-4351874a0a87-plugins-config-volume\") pod \"31ddaa0e-4ac9-419b-a751-4351874a0a87\" (UID: \"31ddaa0e-4ac9-419b-a751-4351874a0a87\") " Nov 02 20:49:01.241790 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.241772 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f2cv\" (UniqueName: \"kubernetes.io/projected/2e4e9d3b-0707-402c-b5ae-7d09880c4f18-kube-api-access-2f2cv\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:01.241891 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.241826 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ddaa0e-4ac9-419b-a751-4351874a0a87-plugins-config-volume" pod "31ddaa0e-4ac9-419b-a751-4351874a0a87" (UID: "31ddaa0e-4ac9-419b-a751-4351874a0a87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:01.243735 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.243717 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ddaa0e-4ac9-419b-a751-4351874a0a87-kube-api-access-b6mfl" pod "31ddaa0e-4ac9-419b-a751-4351874a0a87" (UID: "31ddaa0e-4ac9-419b-a751-4351874a0a87"). InnerVolumeSpecName "kube-api-access-b6mfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:01.342481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.342443 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6mfl\" (UniqueName: \"kubernetes.io/projected/31ddaa0e-4ac9-419b-a751-4351874a0a87-kube-api-access-b6mfl\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:01.342481 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.342473 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ddaa0e-4ac9-419b-a751-4351874a0a87-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:01.412156 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.412117 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:01.699285 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.699248 4971 generic.go:356] "Generic (PLEG): container finished" podID="31ddaa0e-4ac9-419b-a751-4351874a0a87" containerID="b82eb6b49080ec54e0cf9466eea230aa90946ec8c96c843146a44c3f5509bf0b" exitCode=0 Nov 02 20:49:01.699770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.699304 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" Nov 02 20:49:01.699770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.699333 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" event={"ID":"31ddaa0e-4ac9-419b-a751-4351874a0a87","Type":"ContainerDied","Data":"b82eb6b49080ec54e0cf9466eea230aa90946ec8c96c843146a44c3f5509bf0b"} Nov 02 20:49:01.699770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.699431 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs" event={"ID":"31ddaa0e-4ac9-419b-a751-4351874a0a87","Type":"ContainerDied","Data":"179b3f9963e94120934b5cb3bb1de731bf79165455950d225664c60ee3ee727c"} Nov 02 20:49:01.699770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.699464 4971 scope.go:122] "RemoveContainer" containerID="b82eb6b49080ec54e0cf9466eea230aa90946ec8c96c843146a44c3f5509bf0b" Nov 02 20:49:01.701026 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.701003 4971 generic.go:356] "Generic (PLEG): container finished" podID="2e4e9d3b-0707-402c-b5ae-7d09880c4f18" containerID="1b8681c41eda5cbd0e450e42cc6fde6fc6ee63123c51c9b7e8ccb7ddb980ab42" exitCode=0 Nov 02 20:49:01.701168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.701045 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" event={"ID":"2e4e9d3b-0707-402c-b5ae-7d09880c4f18","Type":"ContainerDied","Data":"1b8681c41eda5cbd0e450e42cc6fde6fc6ee63123c51c9b7e8ccb7ddb980ab42"} Nov 02 20:49:01.701168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.701070 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" Nov 02 20:49:01.701168 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.701107 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn" event={"ID":"2e4e9d3b-0707-402c-b5ae-7d09880c4f18","Type":"ContainerDied","Data":"c90ef8c8dffe59412dfab3b3baa934bc7f9b40ba4ce13ade8071ad6459000957"} Nov 02 20:49:01.710540 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.710524 4971 scope.go:122] "RemoveContainer" containerID="b82eb6b49080ec54e0cf9466eea230aa90946ec8c96c843146a44c3f5509bf0b" Nov 02 20:49:01.710796 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:49:01.710776 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82eb6b49080ec54e0cf9466eea230aa90946ec8c96c843146a44c3f5509bf0b\": container with ID starting with b82eb6b49080ec54e0cf9466eea230aa90946ec8c96c843146a44c3f5509bf0b not found: ID does not exist" containerID="b82eb6b49080ec54e0cf9466eea230aa90946ec8c96c843146a44c3f5509bf0b" Nov 02 20:49:01.710865 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.710808 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82eb6b49080ec54e0cf9466eea230aa90946ec8c96c843146a44c3f5509bf0b"} err="failed to get container status \"b82eb6b49080ec54e0cf9466eea230aa90946ec8c96c843146a44c3f5509bf0b\": rpc error: code = NotFound desc = could not find container \"b82eb6b49080ec54e0cf9466eea230aa90946ec8c96c843146a44c3f5509bf0b\": container with ID starting with b82eb6b49080ec54e0cf9466eea230aa90946ec8c96c843146a44c3f5509bf0b not found: ID does not exist" Nov 02 20:49:01.710865 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.710832 4971 scope.go:122] "RemoveContainer" containerID="1b8681c41eda5cbd0e450e42cc6fde6fc6ee63123c51c9b7e8ccb7ddb980ab42" Nov 02 20:49:01.719683 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.719668 4971 scope.go:122] "RemoveContainer" containerID="1b8681c41eda5cbd0e450e42cc6fde6fc6ee63123c51c9b7e8ccb7ddb980ab42" Nov 02 20:49:01.719953 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:49:01.719934 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b8681c41eda5cbd0e450e42cc6fde6fc6ee63123c51c9b7e8ccb7ddb980ab42\": container with ID starting with 1b8681c41eda5cbd0e450e42cc6fde6fc6ee63123c51c9b7e8ccb7ddb980ab42 not found: ID does not exist" containerID="1b8681c41eda5cbd0e450e42cc6fde6fc6ee63123c51c9b7e8ccb7ddb980ab42" Nov 02 20:49:01.720002 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.719961 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8681c41eda5cbd0e450e42cc6fde6fc6ee63123c51c9b7e8ccb7ddb980ab42"} err="failed to get container status \"1b8681c41eda5cbd0e450e42cc6fde6fc6ee63123c51c9b7e8ccb7ddb980ab42\": rpc error: code = NotFound desc = could not find container \"1b8681c41eda5cbd0e450e42cc6fde6fc6ee63123c51c9b7e8ccb7ddb980ab42\": container with ID starting with 1b8681c41eda5cbd0e450e42cc6fde6fc6ee63123c51c9b7e8ccb7ddb980ab42 not found: ID does not exist" Nov 02 20:49:01.855596 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.855567 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn"] Nov 02 20:49:01.867756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.867727 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045025e5-eabc-43de-bc13-681d499ceced" path="/var/lib/kubelet/pods/045025e5-eabc-43de-bc13-681d499ceced/volumes" Nov 02 20:49:01.868140 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.868123 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-58c2e0-7564b5cd86-w8pbn"] Nov 02 20:49:01.963666 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.963571 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs"] Nov 02 20:49:01.969884 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:01.969852 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-58c2e0-77fb77ddd7-gphxs"] Nov 02 20:49:03.340696 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:49:03.340661 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:49:03.865741 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:03.865712 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4e9d3b-0707-402c-b5ae-7d09880c4f18" path="/var/lib/kubelet/pods/2e4e9d3b-0707-402c-b5ae-7d09880c4f18/volumes" Nov 02 20:49:03.866129 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:03.866111 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ddaa0e-4ac9-419b-a751-4351874a0a87" path="/var/lib/kubelet/pods/31ddaa0e-4ac9-419b-a751-4351874a0a87/volumes" Nov 02 20:49:03.877883 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:03.877854 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:04.131242 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:04.131173 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:05.631191 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.631162 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj"] Nov 02 20:49:05.704352 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.704316 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:05.734450 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.734418 4971 generic.go:356] "Generic (PLEG): container finished" podID="dfcede60-ff17-4d6a-b3ab-65dea8834b5e" containerID="c51eee1beefea6d7d5e4ab4838f8302c202dcce84b1f9f51fdff5b5797cb84b8" exitCode=0 Nov 02 20:49:05.734586 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.734489 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" event={"ID":"dfcede60-ff17-4d6a-b3ab-65dea8834b5e","Type":"ContainerDied","Data":"c51eee1beefea6d7d5e4ab4838f8302c202dcce84b1f9f51fdff5b5797cb84b8"} Nov 02 20:49:05.744060 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.743850 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn"] Nov 02 20:49:05.797368 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.797319 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:49:05.886152 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.886115 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-podinfo\") pod \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " Nov 02 20:49:05.886301 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.886211 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-workload-certs\") pod \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " Nov 02 20:49:05.886390 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.886367 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-data\") pod \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " Nov 02 20:49:05.886447 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.886429 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-envoy\") pod \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " Nov 02 20:49:05.886585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.886486 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-token\" (UniqueName: \"kubernetes.io/projected/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-token\") pod \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " Nov 02 20:49:05.886585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.886530 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-workload-certs" pod "dfcede60-ff17-4d6a-b3ab-65dea8834b5e" (UID: "dfcede60-ff17-4d6a-b3ab-65dea8834b5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:05.886708 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.886587 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istiod-ca-cert\") pod \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " Nov 02 20:49:05.886708 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.886643 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-credential-socket\") pod \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " Nov 02 20:49:05.886708 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.886673 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-kube-api-access-f4r55\" (UniqueName: \"kubernetes.io/projected/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-kube-api-access-f4r55\") pod \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " Nov 02 20:49:05.886861 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.886706 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-workload-socket\") pod \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\" (UID: \"dfcede60-ff17-4d6a-b3ab-65dea8834b5e\") " Nov 02 20:49:05.886861 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.886724 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-data" pod "dfcede60-ff17-4d6a-b3ab-65dea8834b5e" (UID: "dfcede60-ff17-4d6a-b3ab-65dea8834b5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:05.886861 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.886852 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-credential-socket" pod "dfcede60-ff17-4d6a-b3ab-65dea8834b5e" (UID: "dfcede60-ff17-4d6a-b3ab-65dea8834b5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:05.887055 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.887036 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-workload-socket" pod "dfcede60-ff17-4d6a-b3ab-65dea8834b5e" (UID: "dfcede60-ff17-4d6a-b3ab-65dea8834b5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:05.887055 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.886957 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istiod-ca-cert" pod "dfcede60-ff17-4d6a-b3ab-65dea8834b5e" (UID: "dfcede60-ff17-4d6a-b3ab-65dea8834b5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:05.887199 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.887097 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:05.887199 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.887119 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:05.887199 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.887133 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:05.888760 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.888736 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-podinfo" pod "dfcede60-ff17-4d6a-b3ab-65dea8834b5e" (UID: "dfcede60-ff17-4d6a-b3ab-65dea8834b5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:49:05.888966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.888947 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-kube-api-access-f4r55" pod "dfcede60-ff17-4d6a-b3ab-65dea8834b5e" (UID: "dfcede60-ff17-4d6a-b3ab-65dea8834b5e"). InnerVolumeSpecName "kube-api-access-f4r55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:05.889222 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.889201 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-token" pod "dfcede60-ff17-4d6a-b3ab-65dea8834b5e" (UID: "dfcede60-ff17-4d6a-b3ab-65dea8834b5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:05.889314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.889211 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-envoy" pod "dfcede60-ff17-4d6a-b3ab-65dea8834b5e" (UID: "dfcede60-ff17-4d6a-b3ab-65dea8834b5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:05.988470 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.988437 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:05.988470 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.988467 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:05.988671 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.988480 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:05.988671 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.988494 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4r55\" (UniqueName: \"kubernetes.io/projected/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-kube-api-access-f4r55\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:05.988671 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.988509 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:05.988671 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:05.988519 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfcede60-ff17-4d6a-b3ab-65dea8834b5e-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:06.185639 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:06.185604 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:06.456067 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:06.455980 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:06.742721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:06.742641 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" event={"ID":"dfcede60-ff17-4d6a-b3ab-65dea8834b5e","Type":"ContainerDied","Data":"4c2b87df56bb605d0a6492a15ea810ed8492f9432cc299e99f7933516095c2b8"} Nov 02 20:49:06.742721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:06.742688 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4" Nov 02 20:49:06.742721 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:06.742709 4971 scope.go:122] "RemoveContainer" containerID="c51eee1beefea6d7d5e4ab4838f8302c202dcce84b1f9f51fdff5b5797cb84b8" Nov 02 20:49:06.785933 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:06.785909 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:06.902113 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:06.902067 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4"] Nov 02 20:49:06.950824 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:06.950793 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-gqmk4"] Nov 02 20:49:07.051894 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:07.051818 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:07.311029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:07.310952 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:07.591124 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:07.591028 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:07.783540 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:07.783512 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5"] Nov 02 20:49:07.783990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:07.783711 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" podUID="7906ad9c-8297-4513-aaf2-8729d580bfa5" containerName="istio-proxy" containerID="cri-o://f663b599c4f90b1b97518b873901265237ffa18d0fab8ebabfb1f99d46614a2a" gracePeriod=30 Nov 02 20:49:07.841157 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:07.841061 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:07.873331 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:07.873294 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfcede60-ff17-4d6a-b3ab-65dea8834b5e" path="/var/lib/kubelet/pods/dfcede60-ff17-4d6a-b3ab-65dea8834b5e/volumes" Nov 02 20:49:08.089381 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.089347 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:08.380576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.380549 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:08.746528 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.746454 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p"] Nov 02 20:49:08.746905 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.746893 4971 cpu_manager.go:414] "containerMap: removing container" podUID="2e4e9d3b-0707-402c-b5ae-7d09880c4f18" containerName="payload-processing" Nov 02 20:49:08.746954 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.746907 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4e9d3b-0707-402c-b5ae-7d09880c4f18" containerName="payload-processing" Nov 02 20:49:08.746954 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.746918 4971 cpu_manager.go:414] "containerMap: removing container" podUID="1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" containerName="istio-proxy" Nov 02 20:49:08.746954 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.746923 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" containerName="istio-proxy" Nov 02 20:49:08.746954 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.746943 4971 cpu_manager.go:414] "containerMap: removing container" podUID="1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" containerName="istio-proxy" Nov 02 20:49:08.746954 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.746948 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" containerName="istio-proxy" Nov 02 20:49:08.747205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.746974 4971 cpu_manager.go:414] "containerMap: removing container" podUID="dfcede60-ff17-4d6a-b3ab-65dea8834b5e" containerName="istio-proxy" Nov 02 20:49:08.747205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.746981 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfcede60-ff17-4d6a-b3ab-65dea8834b5e" containerName="istio-proxy" Nov 02 20:49:08.747205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.746990 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c7c85642-5d6f-49de-8ba4-5cd2a9f543d0" containerName="revoke-keys" Nov 02 20:49:08.747205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.747022 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c85642-5d6f-49de-8ba4-5cd2a9f543d0" containerName="revoke-keys" Nov 02 20:49:08.747205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.747028 4971 cpu_manager.go:414] "containerMap: removing container" podUID="31ddaa0e-4ac9-419b-a751-4351874a0a87" containerName="payload-pre-processing" Nov 02 20:49:08.747205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.747033 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ddaa0e-4ac9-419b-a751-4351874a0a87" containerName="payload-pre-processing" Nov 02 20:49:08.747205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.747124 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" containerName="istio-proxy" Nov 02 20:49:08.747205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.747135 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfcede60-ff17-4d6a-b3ab-65dea8834b5e" containerName="istio-proxy" Nov 02 20:49:08.747205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.747145 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c85642-5d6f-49de-8ba4-5cd2a9f543d0" containerName="revoke-keys" Nov 02 20:49:08.747205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.747152 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ddaa0e-4ac9-419b-a751-4351874a0a87" containerName="payload-pre-processing" Nov 02 20:49:08.747205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.747159 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4e9d3b-0707-402c-b5ae-7d09880c4f18" containerName="payload-processing" Nov 02 20:49:08.747205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.747165 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab9301b-293b-420e-8fb7-ad6eeb6acaa3" containerName="istio-proxy" Nov 02 20:49:08.752124 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.752102 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" Nov 02 20:49:08.801480 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.801458 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:49:08.818474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.818443 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p\" (UID: \"4b63506f-d59c-4b46-81d9-aa1f8968f8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" Nov 02 20:49:08.818628 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.818560 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p\" (UID: \"4b63506f-d59c-4b46-81d9-aa1f8968f8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" Nov 02 20:49:08.818739 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.818719 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxddz\" (UniqueName: \"kubernetes.io/projected/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-kube-api-access-qxddz\") pod \"maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p\" (UID: \"4b63506f-d59c-4b46-81d9-aa1f8968f8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" Nov 02 20:49:08.848253 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.848234 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:49:08.851522 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.851502 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p"] Nov 02 20:49:08.919622 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.919601 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxddz\" (UniqueName: \"kubernetes.io/projected/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-kube-api-access-qxddz\") pod \"maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p\" (UID: \"4b63506f-d59c-4b46-81d9-aa1f8968f8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" Nov 02 20:49:08.919765 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.919644 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p\" (UID: \"4b63506f-d59c-4b46-81d9-aa1f8968f8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" Nov 02 20:49:08.919831 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.919771 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p\" (UID: \"4b63506f-d59c-4b46-81d9-aa1f8968f8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" Nov 02 20:49:08.920253 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.920235 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p\" (UID: \"4b63506f-d59c-4b46-81d9-aa1f8968f8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" Nov 02 20:49:08.922358 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.922331 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p\" (UID: \"4b63506f-d59c-4b46-81d9-aa1f8968f8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" Nov 02 20:49:08.977531 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:08.977512 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxddz\" (UniqueName: \"kubernetes.io/projected/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-kube-api-access-qxddz\") pod \"maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p\" (UID: \"4b63506f-d59c-4b46-81d9-aa1f8968f8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" Nov 02 20:49:09.065510 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:09.065447 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" Nov 02 20:49:09.444936 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:09.444902 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p"] Nov 02 20:49:09.445346 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:49:09.445321 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b63506f_d59c_4b46_81d9_aa1f8968f8c0.slice/crio-316deed1d1c721fe3d2f99ab578fabca58eb7534b54d11015d64006151130297 WatchSource:0}: Error finding container 316deed1d1c721fe3d2f99ab578fabca58eb7534b54d11015d64006151130297: Status 404 returned error can't find the container with id 316deed1d1c721fe3d2f99ab578fabca58eb7534b54d11015d64006151130297 Nov 02 20:49:09.768008 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:09.767972 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" event={"ID":"4b63506f-d59c-4b46-81d9-aa1f8968f8c0","Type":"ContainerStarted","Data":"27204cd8c802a17b84302877e896f61c9a1b3e9671eb7d54a544522fec25cc6f"} Nov 02 20:49:09.768158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:09.768037 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" event={"ID":"4b63506f-d59c-4b46-81d9-aa1f8968f8c0","Type":"ContainerStarted","Data":"316deed1d1c721fe3d2f99ab578fabca58eb7534b54d11015d64006151130297"} Nov 02 20:49:09.878322 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:09.878276 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" podStartSLOduration=1.878262629 podStartE2EDuration="1.878262629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:49:09.876369465 +0000 UTC m=+2078.166025997" watchObservedRunningTime="2026-11-02 20:49:09.878262629 +0000 UTC m=+2078.167919182" Nov 02 20:49:10.082473 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.082409 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq"] Nov 02 20:49:10.086616 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.086598 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq" Nov 02 20:49:10.132842 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.132816 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef9d815-9128-4462-89e7-bca5fe407a55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eda6b36f-cv9kq\" (UID: \"aef9d815-9128-4462-89e7-bca5fe407a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq" Nov 02 20:49:10.132963 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.132860 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b729g\" (UniqueName: \"kubernetes.io/projected/aef9d815-9128-4462-89e7-bca5fe407a55-kube-api-access-b729g\") pod \"maas-api-revoke-sub-eda6b36f-cv9kq\" (UID: \"aef9d815-9128-4462-89e7-bca5fe407a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq" Nov 02 20:49:10.132963 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.132879 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef9d815-9128-4462-89e7-bca5fe407a55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eda6b36f-cv9kq\" (UID: \"aef9d815-9128-4462-89e7-bca5fe407a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq" Nov 02 20:49:10.144201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.144176 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq"] Nov 02 20:49:10.233579 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.233548 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef9d815-9128-4462-89e7-bca5fe407a55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eda6b36f-cv9kq\" (UID: \"aef9d815-9128-4462-89e7-bca5fe407a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq" Nov 02 20:49:10.233717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.233595 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b729g\" (UniqueName: \"kubernetes.io/projected/aef9d815-9128-4462-89e7-bca5fe407a55-kube-api-access-b729g\") pod \"maas-api-revoke-sub-eda6b36f-cv9kq\" (UID: \"aef9d815-9128-4462-89e7-bca5fe407a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq" Nov 02 20:49:10.233717 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.233616 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef9d815-9128-4462-89e7-bca5fe407a55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eda6b36f-cv9kq\" (UID: \"aef9d815-9128-4462-89e7-bca5fe407a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq" Nov 02 20:49:10.234304 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.234280 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef9d815-9128-4462-89e7-bca5fe407a55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eda6b36f-cv9kq\" (UID: \"aef9d815-9128-4462-89e7-bca5fe407a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq" Nov 02 20:49:10.236133 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.236115 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef9d815-9128-4462-89e7-bca5fe407a55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eda6b36f-cv9kq\" (UID: \"aef9d815-9128-4462-89e7-bca5fe407a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq" Nov 02 20:49:10.291850 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.291821 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b729g\" (UniqueName: \"kubernetes.io/projected/aef9d815-9128-4462-89e7-bca5fe407a55-kube-api-access-b729g\") pod \"maas-api-revoke-sub-eda6b36f-cv9kq\" (UID: \"aef9d815-9128-4462-89e7-bca5fe407a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq" Nov 02 20:49:10.400960 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.400934 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq" Nov 02 20:49:10.631905 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.631867 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" podUID="ee36cd67-b157-4fda-bd62-34ba6cd9460e" containerName="payload-processing" containerID="cri-o://b99d29e8e91d614e4104aab2c219fe72b407dd2273f525cd20a6e07546143efd" gracePeriod=25 Nov 02 20:49:10.744144 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.744111 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" podUID="e8daff05-2927-4e7b-95a4-201024607e67" containerName="payload-pre-processing" containerID="cri-o://0cd4f0a112f288becc144823eea65127280484fbb051bbd11ca1e8ec0aaa5b28" gracePeriod=25 Nov 02 20:49:10.768070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.768045 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq"] Nov 02 20:49:10.768372 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:49:10.768347 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef9d815_9128_4462_89e7_bca5fe407a55.slice/crio-c8ebe7c9dff5ed17be2cffca3eb696c8170b1d0000aa11834e21174046644814 WatchSource:0}: Error finding container c8ebe7c9dff5ed17be2cffca3eb696c8170b1d0000aa11834e21174046644814: Status 404 returned error can't find the container with id c8ebe7c9dff5ed17be2cffca3eb696c8170b1d0000aa11834e21174046644814 Nov 02 20:49:10.775927 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.775905 4971 generic.go:356] "Generic (PLEG): container finished" podID="4b63506f-d59c-4b46-81d9-aa1f8968f8c0" containerID="27204cd8c802a17b84302877e896f61c9a1b3e9671eb7d54a544522fec25cc6f" exitCode=0 Nov 02 20:49:10.776018 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:10.775979 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" event={"ID":"4b63506f-d59c-4b46-81d9-aa1f8968f8c0","Type":"ContainerDied","Data":"27204cd8c802a17b84302877e896f61c9a1b3e9671eb7d54a544522fec25cc6f"} Nov 02 20:49:11.069124 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:11.069067 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:11.784541 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:11.784468 4971 generic.go:356] "Generic (PLEG): container finished" podID="aef9d815-9128-4462-89e7-bca5fe407a55" containerID="d2fa78c64cef09118b282d9b6b3cc8bb322fdf3ac620173386425b9372245751" exitCode=0 Nov 02 20:49:11.784688 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:11.784546 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq" event={"ID":"aef9d815-9128-4462-89e7-bca5fe407a55","Type":"ContainerDied","Data":"d2fa78c64cef09118b282d9b6b3cc8bb322fdf3ac620173386425b9372245751"} Nov 02 20:49:11.784688 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:11.784596 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq" event={"ID":"aef9d815-9128-4462-89e7-bca5fe407a55","Type":"ContainerStarted","Data":"c8ebe7c9dff5ed17be2cffca3eb696c8170b1d0000aa11834e21174046644814"} Nov 02 20:49:11.923801 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:11.923772 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" Nov 02 20:49:12.053105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:12.053015 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-maas-api-openshift-ca\") pod \"4b63506f-d59c-4b46-81d9-aa1f8968f8c0\" (UID: \"4b63506f-d59c-4b46-81d9-aa1f8968f8c0\") " Nov 02 20:49:12.053285 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:12.053120 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-kube-api-access-qxddz\" (UniqueName: \"kubernetes.io/projected/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-kube-api-access-qxddz\") pod \"4b63506f-d59c-4b46-81d9-aa1f8968f8c0\" (UID: \"4b63506f-d59c-4b46-81d9-aa1f8968f8c0\") " Nov 02 20:49:12.053285 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:12.053183 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-maas-api-xks-ca\") pod \"4b63506f-d59c-4b46-81d9-aa1f8968f8c0\" (UID: \"4b63506f-d59c-4b46-81d9-aa1f8968f8c0\") " Nov 02 20:49:12.053445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:12.053417 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-maas-api-openshift-ca" pod "4b63506f-d59c-4b46-81d9-aa1f8968f8c0" (UID: "4b63506f-d59c-4b46-81d9-aa1f8968f8c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:12.053604 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:12.053589 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.055289 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:12.055264 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-kube-api-access-qxddz" pod "4b63506f-d59c-4b46-81d9-aa1f8968f8c0" (UID: "4b63506f-d59c-4b46-81d9-aa1f8968f8c0"). InnerVolumeSpecName "kube-api-access-qxddz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:12.055377 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:12.055340 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-maas-api-xks-ca" pod "4b63506f-d59c-4b46-81d9-aa1f8968f8c0" (UID: "4b63506f-d59c-4b46-81d9-aa1f8968f8c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:49:12.154611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:12.154578 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxddz\" (UniqueName: \"kubernetes.io/projected/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-kube-api-access-qxddz\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.154611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:12.154605 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b63506f-d59c-4b46-81d9-aa1f8968f8c0-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.794977 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:12.794952 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" Nov 02 20:49:12.795235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:12.794948 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p" event={"ID":"4b63506f-d59c-4b46-81d9-aa1f8968f8c0","Type":"ContainerDied","Data":"316deed1d1c721fe3d2f99ab578fabca58eb7534b54d11015d64006151130297"} Nov 02 20:49:12.795361 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:12.795272 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="316deed1d1c721fe3d2f99ab578fabca58eb7534b54d11015d64006151130297" Nov 02 20:49:12.936712 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:12.936386 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq" Nov 02 20:49:13.033702 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.033678 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:49:13.064887 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.064863 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef9d815-9128-4462-89e7-bca5fe407a55-kube-api-access-b729g\" (UniqueName: \"kubernetes.io/projected/aef9d815-9128-4462-89e7-bca5fe407a55-kube-api-access-b729g\") pod \"aef9d815-9128-4462-89e7-bca5fe407a55\" (UID: \"aef9d815-9128-4462-89e7-bca5fe407a55\") " Nov 02 20:49:13.065101 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.065055 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aef9d815-9128-4462-89e7-bca5fe407a55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef9d815-9128-4462-89e7-bca5fe407a55-maas-api-openshift-ca\") pod \"aef9d815-9128-4462-89e7-bca5fe407a55\" (UID: \"aef9d815-9128-4462-89e7-bca5fe407a55\") " Nov 02 20:49:13.065215 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.065121 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aef9d815-9128-4462-89e7-bca5fe407a55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef9d815-9128-4462-89e7-bca5fe407a55-maas-api-xks-ca\") pod \"aef9d815-9128-4462-89e7-bca5fe407a55\" (UID: \"aef9d815-9128-4462-89e7-bca5fe407a55\") " Nov 02 20:49:13.065522 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.065499 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef9d815-9128-4462-89e7-bca5fe407a55-maas-api-openshift-ca" pod "aef9d815-9128-4462-89e7-bca5fe407a55" (UID: "aef9d815-9128-4462-89e7-bca5fe407a55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:13.067320 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.067281 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef9d815-9128-4462-89e7-bca5fe407a55-kube-api-access-b729g" pod "aef9d815-9128-4462-89e7-bca5fe407a55" (UID: "aef9d815-9128-4462-89e7-bca5fe407a55"). InnerVolumeSpecName "kube-api-access-b729g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:13.067690 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.067665 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef9d815-9128-4462-89e7-bca5fe407a55-maas-api-xks-ca" pod "aef9d815-9128-4462-89e7-bca5fe407a55" (UID: "aef9d815-9128-4462-89e7-bca5fe407a55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.166416 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-workload-certs\") pod \"7906ad9c-8297-4513-aaf2-8729d580bfa5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.166493 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-credential-socket\") pod \"7906ad9c-8297-4513-aaf2-8729d580bfa5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.166549 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-data\") pod \"7906ad9c-8297-4513-aaf2-8729d580bfa5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.166575 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7906ad9c-8297-4513-aaf2-8729d580bfa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7906ad9c-8297-4513-aaf2-8729d580bfa5-istiod-ca-cert\") pod \"7906ad9c-8297-4513-aaf2-8729d580bfa5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.166629 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-podinfo\") pod \"7906ad9c-8297-4513-aaf2-8729d580bfa5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.166656 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-token\" (UniqueName: \"kubernetes.io/projected/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-token\") pod \"7906ad9c-8297-4513-aaf2-8729d580bfa5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.166686 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-workload-socket\") pod \"7906ad9c-8297-4513-aaf2-8729d580bfa5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.166755 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-workload-certs" pod "7906ad9c-8297-4513-aaf2-8729d580bfa5" (UID: "7906ad9c-8297-4513-aaf2-8729d580bfa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.166778 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-envoy\") pod \"7906ad9c-8297-4513-aaf2-8729d580bfa5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.166808 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7906ad9c-8297-4513-aaf2-8729d580bfa5-kube-api-access-4jkmb\" (UniqueName: \"kubernetes.io/projected/7906ad9c-8297-4513-aaf2-8729d580bfa5-kube-api-access-4jkmb\") pod \"7906ad9c-8297-4513-aaf2-8729d580bfa5\" (UID: \"7906ad9c-8297-4513-aaf2-8729d580bfa5\") " Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.166864 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-credential-socket" pod "7906ad9c-8297-4513-aaf2-8729d580bfa5" (UID: "7906ad9c-8297-4513-aaf2-8729d580bfa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.167065 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-workload-socket" pod "7906ad9c-8297-4513-aaf2-8729d580bfa5" (UID: "7906ad9c-8297-4513-aaf2-8729d580bfa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.167232 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.167251 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef9d815-9128-4462-89e7-bca5fe407a55-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.167266 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef9d815-9128-4462-89e7-bca5fe407a55-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.167281 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b729g\" (UniqueName: \"kubernetes.io/projected/aef9d815-9128-4462-89e7-bca5fe407a55-kube-api-access-b729g\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.167297 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.167303 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.167312 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.170007 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.167928 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7906ad9c-8297-4513-aaf2-8729d580bfa5-istiod-ca-cert" pod "7906ad9c-8297-4513-aaf2-8729d580bfa5" (UID: "7906ad9c-8297-4513-aaf2-8729d580bfa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:13.170007 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.168194 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-data" pod "7906ad9c-8297-4513-aaf2-8729d580bfa5" (UID: "7906ad9c-8297-4513-aaf2-8729d580bfa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:13.170312 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.170277 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-token" pod "7906ad9c-8297-4513-aaf2-8729d580bfa5" (UID: "7906ad9c-8297-4513-aaf2-8729d580bfa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:13.170312 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.170291 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7906ad9c-8297-4513-aaf2-8729d580bfa5-kube-api-access-4jkmb" pod "7906ad9c-8297-4513-aaf2-8729d580bfa5" (UID: "7906ad9c-8297-4513-aaf2-8729d580bfa5"). InnerVolumeSpecName "kube-api-access-4jkmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:13.170634 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.170588 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-envoy" pod "7906ad9c-8297-4513-aaf2-8729d580bfa5" (UID: "7906ad9c-8297-4513-aaf2-8729d580bfa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:13.171371 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.171351 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-podinfo" pod "7906ad9c-8297-4513-aaf2-8729d580bfa5" (UID: "7906ad9c-8297-4513-aaf2-8729d580bfa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:49:13.242371 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.242347 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s"] Nov 02 20:49:13.250679 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.250654 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j"] Nov 02 20:49:13.261804 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.261777 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t"] Nov 02 20:49:13.268889 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.268861 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.268889 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.268886 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.269019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.268897 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.269019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.268908 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jkmb\" (UniqueName: \"kubernetes.io/projected/7906ad9c-8297-4513-aaf2-8729d580bfa5-kube-api-access-4jkmb\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.269019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.268922 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7906ad9c-8297-4513-aaf2-8729d580bfa5-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.269019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.268930 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7906ad9c-8297-4513-aaf2-8729d580bfa5-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.508109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.508033 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:13.803211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.803122 4971 generic.go:356] "Generic (PLEG): container finished" podID="7906ad9c-8297-4513-aaf2-8729d580bfa5" containerID="f663b599c4f90b1b97518b873901265237ffa18d0fab8ebabfb1f99d46614a2a" exitCode=0 Nov 02 20:49:13.803211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.803191 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" Nov 02 20:49:13.803417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.803207 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" event={"ID":"7906ad9c-8297-4513-aaf2-8729d580bfa5","Type":"ContainerDied","Data":"f663b599c4f90b1b97518b873901265237ffa18d0fab8ebabfb1f99d46614a2a"} Nov 02 20:49:13.803417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.803291 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5" event={"ID":"7906ad9c-8297-4513-aaf2-8729d580bfa5","Type":"ContainerDied","Data":"7d04e449bdaa1d91042cecbb678f79680991588a2c29dc669ed5043ea2635acc"} Nov 02 20:49:13.803417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.803316 4971 scope.go:122] "RemoveContainer" containerID="f663b599c4f90b1b97518b873901265237ffa18d0fab8ebabfb1f99d46614a2a" Nov 02 20:49:13.805260 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.805234 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq" Nov 02 20:49:13.805379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.805257 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq" event={"ID":"aef9d815-9128-4462-89e7-bca5fe407a55","Type":"ContainerDied","Data":"c8ebe7c9dff5ed17be2cffca3eb696c8170b1d0000aa11834e21174046644814"} Nov 02 20:49:13.805379 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.805298 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8ebe7c9dff5ed17be2cffca3eb696c8170b1d0000aa11834e21174046644814" Nov 02 20:49:13.814856 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.814839 4971 scope.go:122] "RemoveContainer" containerID="78cc9c5d21c26ab1e80a89b299840e5a902490637285aadc77ece1b75fc15b90" Nov 02 20:49:13.824151 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.824131 4971 scope.go:122] "RemoveContainer" containerID="f663b599c4f90b1b97518b873901265237ffa18d0fab8ebabfb1f99d46614a2a" Nov 02 20:49:13.824423 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:49:13.824405 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f663b599c4f90b1b97518b873901265237ffa18d0fab8ebabfb1f99d46614a2a\": container with ID starting with f663b599c4f90b1b97518b873901265237ffa18d0fab8ebabfb1f99d46614a2a not found: ID does not exist" containerID="f663b599c4f90b1b97518b873901265237ffa18d0fab8ebabfb1f99d46614a2a" Nov 02 20:49:13.824468 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.824430 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f663b599c4f90b1b97518b873901265237ffa18d0fab8ebabfb1f99d46614a2a"} err="failed to get container status \"f663b599c4f90b1b97518b873901265237ffa18d0fab8ebabfb1f99d46614a2a\": rpc error: code = NotFound desc = could not find container \"f663b599c4f90b1b97518b873901265237ffa18d0fab8ebabfb1f99d46614a2a\": container with ID starting with f663b599c4f90b1b97518b873901265237ffa18d0fab8ebabfb1f99d46614a2a not found: ID does not exist" Nov 02 20:49:13.824468 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.824447 4971 scope.go:122] "RemoveContainer" containerID="78cc9c5d21c26ab1e80a89b299840e5a902490637285aadc77ece1b75fc15b90" Nov 02 20:49:13.824702 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:49:13.824685 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78cc9c5d21c26ab1e80a89b299840e5a902490637285aadc77ece1b75fc15b90\": container with ID starting with 78cc9c5d21c26ab1e80a89b299840e5a902490637285aadc77ece1b75fc15b90 not found: ID does not exist" containerID="78cc9c5d21c26ab1e80a89b299840e5a902490637285aadc77ece1b75fc15b90" Nov 02 20:49:13.824745 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.824708 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78cc9c5d21c26ab1e80a89b299840e5a902490637285aadc77ece1b75fc15b90"} err="failed to get container status \"78cc9c5d21c26ab1e80a89b299840e5a902490637285aadc77ece1b75fc15b90\": rpc error: code = NotFound desc = could not find container \"78cc9c5d21c26ab1e80a89b299840e5a902490637285aadc77ece1b75fc15b90\": container with ID starting with 78cc9c5d21c26ab1e80a89b299840e5a902490637285aadc77ece1b75fc15b90 not found: ID does not exist" Nov 02 20:49:13.959542 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:13.959511 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5"] Nov 02 20:49:14.007746 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:14.007717 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-58c2e0-openshift-default-59d78758cc-shmt5"] Nov 02 20:49:15.866510 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:15.866478 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7906ad9c-8297-4513-aaf2-8729d580bfa5" path="/var/lib/kubelet/pods/7906ad9c-8297-4513-aaf2-8729d580bfa5/volumes" Nov 02 20:49:16.080211 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:16.080174 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:19.616329 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:19.616292 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" podUID="e8daff05-2927-4e7b-95a4-201024607e67" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:49:19.616845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:19.616290 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" podUID="ee36cd67-b157-4fda-bd62-34ba6cd9460e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:49:20.387435 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.387388 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69"] Nov 02 20:49:20.388107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.388052 4971 cpu_manager.go:414] "containerMap: removing container" podUID="7906ad9c-8297-4513-aaf2-8729d580bfa5" containerName="istio-proxy" Nov 02 20:49:20.388107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.388098 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7906ad9c-8297-4513-aaf2-8729d580bfa5" containerName="istio-proxy" Nov 02 20:49:20.388313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.388116 4971 cpu_manager.go:414] "containerMap: removing container" podUID="4b63506f-d59c-4b46-81d9-aa1f8968f8c0" containerName="revoke-keys" Nov 02 20:49:20.388313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.388123 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b63506f-d59c-4b46-81d9-aa1f8968f8c0" containerName="revoke-keys" Nov 02 20:49:20.388313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.388162 4971 cpu_manager.go:414] "containerMap: removing container" podUID="aef9d815-9128-4462-89e7-bca5fe407a55" containerName="revoke-keys" Nov 02 20:49:20.388313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.388171 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aef9d815-9128-4462-89e7-bca5fe407a55" containerName="revoke-keys" Nov 02 20:49:20.388313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.388185 4971 cpu_manager.go:414] "containerMap: removing container" podUID="7906ad9c-8297-4513-aaf2-8729d580bfa5" containerName="istio-proxy" Nov 02 20:49:20.388313 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.388194 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7906ad9c-8297-4513-aaf2-8729d580bfa5" containerName="istio-proxy" Nov 02 20:49:20.388581 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.388348 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="aef9d815-9128-4462-89e7-bca5fe407a55" containerName="revoke-keys" Nov 02 20:49:20.388581 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.388364 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="7906ad9c-8297-4513-aaf2-8729d580bfa5" containerName="istio-proxy" Nov 02 20:49:20.388581 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.388380 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="7906ad9c-8297-4513-aaf2-8729d580bfa5" containerName="istio-proxy" Nov 02 20:49:20.388581 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.388391 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b63506f-d59c-4b46-81d9-aa1f8968f8c0" containerName="revoke-keys" Nov 02 20:49:20.394152 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.394131 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" Nov 02 20:49:20.444359 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.444321 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:49:20.444502 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.444380 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d"] Nov 02 20:49:20.446759 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.446737 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:49:20.448530 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.448511 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" Nov 02 20:49:20.491209 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.491182 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69"] Nov 02 20:49:20.516323 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.516286 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d"] Nov 02 20:49:20.539658 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.539626 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlgh4\" (UniqueName: \"kubernetes.io/projected/b46adb19-9afd-4671-94fe-4a5f6b965dfb-kube-api-access-nlgh4\") pod \"maas-api-revoke-sub-a7f91883-p7k69\" (UID: \"b46adb19-9afd-4671-94fe-4a5f6b965dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" Nov 02 20:49:20.539803 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.539670 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e88e7eef-5eb8-4274-ad74-4c691a426260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01bcb75f-qfb8d\" (UID: \"e88e7eef-5eb8-4274-ad74-4c691a426260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" Nov 02 20:49:20.539803 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.539767 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46adb19-9afd-4671-94fe-4a5f6b965dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7f91883-p7k69\" (UID: \"b46adb19-9afd-4671-94fe-4a5f6b965dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" Nov 02 20:49:20.539889 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.539870 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e88e7eef-5eb8-4274-ad74-4c691a426260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01bcb75f-qfb8d\" (UID: \"e88e7eef-5eb8-4274-ad74-4c691a426260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" Nov 02 20:49:20.539947 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.539927 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4f8j\" (UniqueName: \"kubernetes.io/projected/e88e7eef-5eb8-4274-ad74-4c691a426260-kube-api-access-r4f8j\") pod \"maas-api-revoke-sub-01bcb75f-qfb8d\" (UID: \"e88e7eef-5eb8-4274-ad74-4c691a426260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" Nov 02 20:49:20.540064 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.540041 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46adb19-9afd-4671-94fe-4a5f6b965dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7f91883-p7k69\" (UID: \"b46adb19-9afd-4671-94fe-4a5f6b965dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" Nov 02 20:49:20.652730 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.652635 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46adb19-9afd-4671-94fe-4a5f6b965dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7f91883-p7k69\" (UID: \"b46adb19-9afd-4671-94fe-4a5f6b965dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" Nov 02 20:49:20.667375 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.667347 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e88e7eef-5eb8-4274-ad74-4c691a426260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01bcb75f-qfb8d\" (UID: \"e88e7eef-5eb8-4274-ad74-4c691a426260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" Nov 02 20:49:20.668062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.668038 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4f8j\" (UniqueName: \"kubernetes.io/projected/e88e7eef-5eb8-4274-ad74-4c691a426260-kube-api-access-r4f8j\") pod \"maas-api-revoke-sub-01bcb75f-qfb8d\" (UID: \"e88e7eef-5eb8-4274-ad74-4c691a426260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" Nov 02 20:49:20.668664 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.668644 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46adb19-9afd-4671-94fe-4a5f6b965dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7f91883-p7k69\" (UID: \"b46adb19-9afd-4671-94fe-4a5f6b965dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" Nov 02 20:49:20.669601 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.669574 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlgh4\" (UniqueName: \"kubernetes.io/projected/b46adb19-9afd-4671-94fe-4a5f6b965dfb-kube-api-access-nlgh4\") pod \"maas-api-revoke-sub-a7f91883-p7k69\" (UID: \"b46adb19-9afd-4671-94fe-4a5f6b965dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" Nov 02 20:49:20.674400 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.669504 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46adb19-9afd-4671-94fe-4a5f6b965dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7f91883-p7k69\" (UID: \"b46adb19-9afd-4671-94fe-4a5f6b965dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" Nov 02 20:49:20.674561 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.674541 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e88e7eef-5eb8-4274-ad74-4c691a426260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01bcb75f-qfb8d\" (UID: \"e88e7eef-5eb8-4274-ad74-4c691a426260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" Nov 02 20:49:20.675461 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.675344 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e88e7eef-5eb8-4274-ad74-4c691a426260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01bcb75f-qfb8d\" (UID: \"e88e7eef-5eb8-4274-ad74-4c691a426260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" Nov 02 20:49:20.676102 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.676058 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46adb19-9afd-4671-94fe-4a5f6b965dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7f91883-p7k69\" (UID: \"b46adb19-9afd-4671-94fe-4a5f6b965dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" Nov 02 20:49:20.682396 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.679602 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e88e7eef-5eb8-4274-ad74-4c691a426260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01bcb75f-qfb8d\" (UID: \"e88e7eef-5eb8-4274-ad74-4c691a426260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" Nov 02 20:49:20.798145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.785581 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:20.798145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.786637 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4f8j\" (UniqueName: \"kubernetes.io/projected/e88e7eef-5eb8-4274-ad74-4c691a426260-kube-api-access-r4f8j\") pod \"maas-api-revoke-sub-01bcb75f-qfb8d\" (UID: \"e88e7eef-5eb8-4274-ad74-4c691a426260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" Nov 02 20:49:20.800663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:20.800624 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlgh4\" (UniqueName: \"kubernetes.io/projected/b46adb19-9afd-4671-94fe-4a5f6b965dfb-kube-api-access-nlgh4\") pod \"maas-api-revoke-sub-a7f91883-p7k69\" (UID: \"b46adb19-9afd-4671-94fe-4a5f6b965dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" Nov 02 20:49:21.008092 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:21.007995 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" Nov 02 20:49:21.045741 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:21.045699 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:21.066108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:21.066053 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" Nov 02 20:49:21.393383 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:21.393357 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69"] Nov 02 20:49:21.394752 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:49:21.394725 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46adb19_9afd_4671_94fe_4a5f6b965dfb.slice/crio-d6d2fc7485207bd0fe02fc5c2d1973d9e4b103551d881f7e8e843ff37ed09972 WatchSource:0}: Error finding container d6d2fc7485207bd0fe02fc5c2d1973d9e4b103551d881f7e8e843ff37ed09972: Status 404 returned error can't find the container with id d6d2fc7485207bd0fe02fc5c2d1973d9e4b103551d881f7e8e843ff37ed09972 Nov 02 20:49:21.488173 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:21.488138 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d"] Nov 02 20:49:21.489593 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:49:21.489556 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88e7eef_5eb8_4274_ad74_4c691a426260.slice/crio-ec61ae54189f6d34daf2e59c20ae7361ccae8ef4fed208452fbfa01a8f69f8ef WatchSource:0}: Error finding container ec61ae54189f6d34daf2e59c20ae7361ccae8ef4fed208452fbfa01a8f69f8ef: Status 404 returned error can't find the container with id ec61ae54189f6d34daf2e59c20ae7361ccae8ef4fed208452fbfa01a8f69f8ef Nov 02 20:49:21.873093 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:21.873038 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" event={"ID":"b46adb19-9afd-4671-94fe-4a5f6b965dfb","Type":"ContainerStarted","Data":"b54adab38f5f760838845a306e3862fd7b67f3e6a948a0cc3433f168086a5037"} Nov 02 20:49:21.873616 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:21.873126 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" event={"ID":"b46adb19-9afd-4671-94fe-4a5f6b965dfb","Type":"ContainerStarted","Data":"d6d2fc7485207bd0fe02fc5c2d1973d9e4b103551d881f7e8e843ff37ed09972"} Nov 02 20:49:21.874816 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:21.874777 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" event={"ID":"e88e7eef-5eb8-4274-ad74-4c691a426260","Type":"ContainerStarted","Data":"b16cf3ad34dd26baf821edaa14b9b678629f86c49dea5b36b1625a7dc25e8d62"} Nov 02 20:49:21.874950 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:21.874827 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" event={"ID":"e88e7eef-5eb8-4274-ad74-4c691a426260","Type":"ContainerStarted","Data":"ec61ae54189f6d34daf2e59c20ae7361ccae8ef4fed208452fbfa01a8f69f8ef"} Nov 02 20:49:21.985255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:21.985138 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" podStartSLOduration=1.9851242070000001 podStartE2EDuration="1.985124207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:49:21.982167728 +0000 UTC m=+2090.271824258" watchObservedRunningTime="2026-11-02 20:49:21.985124207 +0000 UTC m=+2090.274780734" Nov 02 20:49:22.093506 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:22.093435 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" podStartSLOduration=2.093415144 podStartE2EDuration="2.093415144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:49:22.090948323 +0000 UTC m=+2090.380604851" watchObservedRunningTime="2026-11-02 20:49:22.093415144 +0000 UTC m=+2090.383071672" Nov 02 20:49:22.882742 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:22.882701 4971 generic.go:356] "Generic (PLEG): container finished" podID="b46adb19-9afd-4671-94fe-4a5f6b965dfb" containerID="b54adab38f5f760838845a306e3862fd7b67f3e6a948a0cc3433f168086a5037" exitCode=0 Nov 02 20:49:22.883247 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:22.882783 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" event={"ID":"b46adb19-9afd-4671-94fe-4a5f6b965dfb","Type":"ContainerDied","Data":"b54adab38f5f760838845a306e3862fd7b67f3e6a948a0cc3433f168086a5037"} Nov 02 20:49:22.884636 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:22.884613 4971 generic.go:356] "Generic (PLEG): container finished" podID="e88e7eef-5eb8-4274-ad74-4c691a426260" containerID="b16cf3ad34dd26baf821edaa14b9b678629f86c49dea5b36b1625a7dc25e8d62" exitCode=0 Nov 02 20:49:22.884757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:22.884670 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" event={"ID":"e88e7eef-5eb8-4274-ad74-4c691a426260","Type":"ContainerDied","Data":"b16cf3ad34dd26baf821edaa14b9b678629f86c49dea5b36b1625a7dc25e8d62"} Nov 02 20:49:24.055880 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.055855 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" Nov 02 20:49:24.059570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.059551 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" Nov 02 20:49:24.124454 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.124431 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e88e7eef-5eb8-4274-ad74-4c691a426260-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e88e7eef-5eb8-4274-ad74-4c691a426260-maas-api-xks-ca\") pod \"e88e7eef-5eb8-4274-ad74-4c691a426260\" (UID: \"e88e7eef-5eb8-4274-ad74-4c691a426260\") " Nov 02 20:49:24.124629 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.124506 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88e7eef-5eb8-4274-ad74-4c691a426260-kube-api-access-r4f8j\" (UniqueName: \"kubernetes.io/projected/e88e7eef-5eb8-4274-ad74-4c691a426260-kube-api-access-r4f8j\") pod \"e88e7eef-5eb8-4274-ad74-4c691a426260\" (UID: \"e88e7eef-5eb8-4274-ad74-4c691a426260\") " Nov 02 20:49:24.124629 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.124538 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e88e7eef-5eb8-4274-ad74-4c691a426260-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e88e7eef-5eb8-4274-ad74-4c691a426260-maas-api-openshift-ca\") pod \"e88e7eef-5eb8-4274-ad74-4c691a426260\" (UID: \"e88e7eef-5eb8-4274-ad74-4c691a426260\") " Nov 02 20:49:24.124629 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.124616 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b46adb19-9afd-4671-94fe-4a5f6b965dfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46adb19-9afd-4671-94fe-4a5f6b965dfb-maas-api-xks-ca\") pod \"b46adb19-9afd-4671-94fe-4a5f6b965dfb\" (UID: \"b46adb19-9afd-4671-94fe-4a5f6b965dfb\") " Nov 02 20:49:24.124812 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.124674 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46adb19-9afd-4671-94fe-4a5f6b965dfb-kube-api-access-nlgh4\" (UniqueName: \"kubernetes.io/projected/b46adb19-9afd-4671-94fe-4a5f6b965dfb-kube-api-access-nlgh4\") pod \"b46adb19-9afd-4671-94fe-4a5f6b965dfb\" (UID: \"b46adb19-9afd-4671-94fe-4a5f6b965dfb\") " Nov 02 20:49:24.124893 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.124864 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46adb19-9afd-4671-94fe-4a5f6b965dfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46adb19-9afd-4671-94fe-4a5f6b965dfb-maas-api-openshift-ca\") pod \"b46adb19-9afd-4671-94fe-4a5f6b965dfb\" (UID: \"b46adb19-9afd-4671-94fe-4a5f6b965dfb\") " Nov 02 20:49:24.125017 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.124978 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e88e7eef-5eb8-4274-ad74-4c691a426260-maas-api-openshift-ca" pod "e88e7eef-5eb8-4274-ad74-4c691a426260" (UID: "e88e7eef-5eb8-4274-ad74-4c691a426260"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:24.125398 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.125340 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46adb19-9afd-4671-94fe-4a5f6b965dfb-maas-api-openshift-ca" pod "b46adb19-9afd-4671-94fe-4a5f6b965dfb" (UID: "b46adb19-9afd-4671-94fe-4a5f6b965dfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:24.125601 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.125572 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46adb19-9afd-4671-94fe-4a5f6b965dfb-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:24.125686 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.125602 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e88e7eef-5eb8-4274-ad74-4c691a426260-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:24.126753 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.126729 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e88e7eef-5eb8-4274-ad74-4c691a426260-maas-api-xks-ca" pod "e88e7eef-5eb8-4274-ad74-4c691a426260" (UID: "e88e7eef-5eb8-4274-ad74-4c691a426260"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:49:24.126856 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.126841 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88e7eef-5eb8-4274-ad74-4c691a426260-kube-api-access-r4f8j" pod "e88e7eef-5eb8-4274-ad74-4c691a426260" (UID: "e88e7eef-5eb8-4274-ad74-4c691a426260"). InnerVolumeSpecName "kube-api-access-r4f8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:24.127430 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.127400 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b46adb19-9afd-4671-94fe-4a5f6b965dfb-maas-api-xks-ca" pod "b46adb19-9afd-4671-94fe-4a5f6b965dfb" (UID: "b46adb19-9afd-4671-94fe-4a5f6b965dfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:49:24.127516 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.127441 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46adb19-9afd-4671-94fe-4a5f6b965dfb-kube-api-access-nlgh4" pod "b46adb19-9afd-4671-94fe-4a5f6b965dfb" (UID: "b46adb19-9afd-4671-94fe-4a5f6b965dfb"). InnerVolumeSpecName "kube-api-access-nlgh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:24.226689 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.226624 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlgh4\" (UniqueName: \"kubernetes.io/projected/b46adb19-9afd-4671-94fe-4a5f6b965dfb-kube-api-access-nlgh4\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:24.226689 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.226652 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e88e7eef-5eb8-4274-ad74-4c691a426260-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:24.226689 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.226662 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4f8j\" (UniqueName: \"kubernetes.io/projected/e88e7eef-5eb8-4274-ad74-4c691a426260-kube-api-access-r4f8j\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:24.226689 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.226671 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46adb19-9afd-4671-94fe-4a5f6b965dfb-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:24.898526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.898452 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" Nov 02 20:49:24.898526 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.898491 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69" event={"ID":"b46adb19-9afd-4671-94fe-4a5f6b965dfb","Type":"ContainerDied","Data":"d6d2fc7485207bd0fe02fc5c2d1973d9e4b103551d881f7e8e843ff37ed09972"} Nov 02 20:49:24.898762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.898534 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6d2fc7485207bd0fe02fc5c2d1973d9e4b103551d881f7e8e843ff37ed09972" Nov 02 20:49:24.899860 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.899830 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" Nov 02 20:49:24.899993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.899869 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d" event={"ID":"e88e7eef-5eb8-4274-ad74-4c691a426260","Type":"ContainerDied","Data":"ec61ae54189f6d34daf2e59c20ae7361ccae8ef4fed208452fbfa01a8f69f8ef"} Nov 02 20:49:24.899993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:24.899899 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec61ae54189f6d34daf2e59c20ae7361ccae8ef4fed208452fbfa01a8f69f8ef" Nov 02 20:49:28.263864 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.263811 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" containerID="cri-o://1b44310cacce629cddf37549ff26e06391314064c0e9425a6f435f1b5b9e6d0d" gracePeriod=45 Nov 02 20:49:28.278527 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.278485 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" containerID="cri-o://cde583b432e41744fe7e87c7dfbd2f4ee8c394fe5b25f89d4e0e5b16d6026534" gracePeriod=45 Nov 02 20:49:28.281149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.281062 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" containerID="cri-o://d2607e7731514643c5be9110331e89d06781e9f7f4595bde90be8e0d7307a008" gracePeriod=45 Nov 02 20:49:28.303856 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.303826 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": dial tcp 10.132.0.69:8000: connect: connection refused" Nov 02 20:49:28.306154 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.306132 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": dial tcp 10.132.0.71:8000: connect: connection refused" Nov 02 20:49:28.582766 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.582732 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:49:28.590325 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.590302 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:49:28.598510 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.598489 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:49:28.666682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.666655 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4486f93b-6513-4998-a2c7-d55f05805ee1-kube-api-access-ht92t\" (UniqueName: \"kubernetes.io/projected/4486f93b-6513-4998-a2c7-d55f05805ee1-kube-api-access-ht92t\") pod \"4486f93b-6513-4998-a2c7-d55f05805ee1\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " Nov 02 20:49:28.666846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.666709 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cfa705d-47bf-4d68-8f92-a9132c911b6b-kube-api-access-6bf9x\" (UniqueName: \"kubernetes.io/projected/0cfa705d-47bf-4d68-8f92-a9132c911b6b-kube-api-access-6bf9x\") pod \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " Nov 02 20:49:28.666846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.666734 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-dshm\") pod \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " Nov 02 20:49:28.666846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.666762 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-model-cache\") pod \"4486f93b-6513-4998-a2c7-d55f05805ee1\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " Nov 02 20:49:28.666846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.666798 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-dshm\") pod \"4486f93b-6513-4998-a2c7-d55f05805ee1\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " Nov 02 20:49:28.666846 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.666835 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4486f93b-6513-4998-a2c7-d55f05805ee1-tls-certs\" (UniqueName: \"kubernetes.io/secret/4486f93b-6513-4998-a2c7-d55f05805ee1-tls-certs\") pod \"4486f93b-6513-4998-a2c7-d55f05805ee1\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " Nov 02 20:49:28.667166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.666873 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-model-cache\") pod \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " Nov 02 20:49:28.667166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.666953 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-home\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-home\") pod \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " Nov 02 20:49:28.667166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.666995 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-home\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-home\") pod \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " Nov 02 20:49:28.667166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667037 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-model-cache\") pod \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " Nov 02 20:49:28.667166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667038 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-model-cache" pod "4486f93b-6513-4998-a2c7-d55f05805ee1" (UID: "4486f93b-6513-4998-a2c7-d55f05805ee1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:28.667166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667073 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-tmp-dir\") pod \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " Nov 02 20:49:28.667166 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667155 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-tmp-dir\") pod \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " Nov 02 20:49:28.667547 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667227 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-home\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-home\") pod \"4486f93b-6513-4998-a2c7-d55f05805ee1\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " Nov 02 20:49:28.667547 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667264 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-tls-certs\") pod \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " Nov 02 20:49:28.667547 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667280 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-home" pod "aa6f0ad6-43f0-484a-9f00-55f7dcef5698" (UID: "aa6f0ad6-43f0-484a-9f00-55f7dcef5698"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:28.667547 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667292 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-dshm\") pod \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " Nov 02 20:49:28.667547 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667338 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cfa705d-47bf-4d68-8f92-a9132c911b6b-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cfa705d-47bf-4d68-8f92-a9132c911b6b-tls-certs\") pod \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\" (UID: \"0cfa705d-47bf-4d68-8f92-a9132c911b6b\") " Nov 02 20:49:28.667547 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667373 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-tmp-dir\") pod \"4486f93b-6513-4998-a2c7-d55f05805ee1\" (UID: \"4486f93b-6513-4998-a2c7-d55f05805ee1\") " Nov 02 20:49:28.667547 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667420 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-kube-api-access-gfcs2\" (UniqueName: \"kubernetes.io/projected/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-kube-api-access-gfcs2\") pod \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\" (UID: \"aa6f0ad6-43f0-484a-9f00-55f7dcef5698\") " Nov 02 20:49:28.667547 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667447 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-tmp-dir" pod "0cfa705d-47bf-4d68-8f92-a9132c911b6b" (UID: "0cfa705d-47bf-4d68-8f92-a9132c911b6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:28.667986 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667667 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-model-cache" pod "0cfa705d-47bf-4d68-8f92-a9132c911b6b" (UID: "0cfa705d-47bf-4d68-8f92-a9132c911b6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:28.667986 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667840 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-home" pod "0cfa705d-47bf-4d68-8f92-a9132c911b6b" (UID: "0cfa705d-47bf-4d68-8f92-a9132c911b6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:28.667986 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667879 4971 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-home\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.667986 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667897 4971 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-model-cache\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.667986 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667913 4971 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-tmp-dir\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.667986 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.667929 4971 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-model-cache\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.668504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.668068 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-model-cache" pod "aa6f0ad6-43f0-484a-9f00-55f7dcef5698" (UID: "aa6f0ad6-43f0-484a-9f00-55f7dcef5698"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:28.668504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.668443 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-tmp-dir" pod "4486f93b-6513-4998-a2c7-d55f05805ee1" (UID: "4486f93b-6513-4998-a2c7-d55f05805ee1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:28.668504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.668473 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-tmp-dir" pod "aa6f0ad6-43f0-484a-9f00-55f7dcef5698" (UID: "aa6f0ad6-43f0-484a-9f00-55f7dcef5698"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:28.668696 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.668647 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-home" pod "4486f93b-6513-4998-a2c7-d55f05805ee1" (UID: "4486f93b-6513-4998-a2c7-d55f05805ee1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:28.670190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.670152 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-dshm" pod "4486f93b-6513-4998-a2c7-d55f05805ee1" (UID: "4486f93b-6513-4998-a2c7-d55f05805ee1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:28.670895 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.670870 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-dshm" pod "0cfa705d-47bf-4d68-8f92-a9132c911b6b" (UID: "0cfa705d-47bf-4d68-8f92-a9132c911b6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:28.670995 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.670961 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-kube-api-access-gfcs2" pod "aa6f0ad6-43f0-484a-9f00-55f7dcef5698" (UID: "aa6f0ad6-43f0-484a-9f00-55f7dcef5698"). InnerVolumeSpecName "kube-api-access-gfcs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:28.671070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.671048 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-tls-certs" pod "aa6f0ad6-43f0-484a-9f00-55f7dcef5698" (UID: "aa6f0ad6-43f0-484a-9f00-55f7dcef5698"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:49:28.671599 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.671575 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-dshm" pod "aa6f0ad6-43f0-484a-9f00-55f7dcef5698" (UID: "aa6f0ad6-43f0-484a-9f00-55f7dcef5698"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:28.672245 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.672222 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cfa705d-47bf-4d68-8f92-a9132c911b6b-kube-api-access-6bf9x" pod "0cfa705d-47bf-4d68-8f92-a9132c911b6b" (UID: "0cfa705d-47bf-4d68-8f92-a9132c911b6b"). InnerVolumeSpecName "kube-api-access-6bf9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:28.672357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.672262 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4486f93b-6513-4998-a2c7-d55f05805ee1-tls-certs" pod "4486f93b-6513-4998-a2c7-d55f05805ee1" (UID: "4486f93b-6513-4998-a2c7-d55f05805ee1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:49:28.672497 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.672472 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4486f93b-6513-4998-a2c7-d55f05805ee1-kube-api-access-ht92t" pod "4486f93b-6513-4998-a2c7-d55f05805ee1" (UID: "4486f93b-6513-4998-a2c7-d55f05805ee1"). InnerVolumeSpecName "kube-api-access-ht92t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:28.672807 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.672785 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cfa705d-47bf-4d68-8f92-a9132c911b6b-tls-certs" pod "0cfa705d-47bf-4d68-8f92-a9132c911b6b" (UID: "0cfa705d-47bf-4d68-8f92-a9132c911b6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:49:28.769429 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.769396 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bf9x\" (UniqueName: \"kubernetes.io/projected/0cfa705d-47bf-4d68-8f92-a9132c911b6b-kube-api-access-6bf9x\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.769429 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.769428 4971 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-dshm\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.769632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.769438 4971 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-dshm\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.769632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.769446 4971 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4486f93b-6513-4998-a2c7-d55f05805ee1-tls-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.769632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.769455 4971 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-model-cache\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.769632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.769463 4971 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cfa705d-47bf-4d68-8f92-a9132c911b6b-home\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.769632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.769471 4971 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-tmp-dir\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.769632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.769478 4971 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-home\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.769632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.769485 4971 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-tls-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.769632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.769492 4971 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-dshm\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.769632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.769500 4971 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cfa705d-47bf-4d68-8f92-a9132c911b6b-tls-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.769632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.769508 4971 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4486f93b-6513-4998-a2c7-d55f05805ee1-tmp-dir\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.769632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.769516 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfcs2\" (UniqueName: \"kubernetes.io/projected/aa6f0ad6-43f0-484a-9f00-55f7dcef5698-kube-api-access-gfcs2\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.769632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.769524 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht92t\" (UniqueName: \"kubernetes.io/projected/4486f93b-6513-4998-a2c7-d55f05805ee1-kube-api-access-ht92t\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.929881 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.929843 4971 generic.go:356] "Generic (PLEG): container finished" podID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerID="1b44310cacce629cddf37549ff26e06391314064c0e9425a6f435f1b5b9e6d0d" exitCode=0 Nov 02 20:49:28.930105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.929877 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" event={"ID":"0cfa705d-47bf-4d68-8f92-a9132c911b6b","Type":"ContainerDied","Data":"1b44310cacce629cddf37549ff26e06391314064c0e9425a6f435f1b5b9e6d0d"} Nov 02 20:49:28.930105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.929923 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" event={"ID":"0cfa705d-47bf-4d68-8f92-a9132c911b6b","Type":"ContainerDied","Data":"8f2c662d1c73da4a7d6926b5dd097bb814987fb71c379c64143c228253a69787"} Nov 02 20:49:28.930105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.929942 4971 scope.go:122] "RemoveContainer" containerID="1b44310cacce629cddf37549ff26e06391314064c0e9425a6f435f1b5b9e6d0d" Nov 02 20:49:28.930105 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.929892 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s" Nov 02 20:49:28.932802 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.932779 4971 generic.go:356] "Generic (PLEG): container finished" podID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerID="d2607e7731514643c5be9110331e89d06781e9f7f4595bde90be8e0d7307a008" exitCode=0 Nov 02 20:49:28.932904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.932866 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" event={"ID":"aa6f0ad6-43f0-484a-9f00-55f7dcef5698","Type":"ContainerDied","Data":"d2607e7731514643c5be9110331e89d06781e9f7f4595bde90be8e0d7307a008"} Nov 02 20:49:28.932904 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.932892 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" event={"ID":"aa6f0ad6-43f0-484a-9f00-55f7dcef5698","Type":"ContainerDied","Data":"998ea4add1ed641c8479edff91e46ae93f896605af32a0c7e84190b6e4afb7e3"} Nov 02 20:49:28.932983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.932930 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t" Nov 02 20:49:28.939711 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.939683 4971 generic.go:356] "Generic (PLEG): container finished" podID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerID="cde583b432e41744fe7e87c7dfbd2f4ee8c394fe5b25f89d4e0e5b16d6026534" exitCode=0 Nov 02 20:49:28.939856 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.939837 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" event={"ID":"4486f93b-6513-4998-a2c7-d55f05805ee1","Type":"ContainerDied","Data":"cde583b432e41744fe7e87c7dfbd2f4ee8c394fe5b25f89d4e0e5b16d6026534"} Nov 02 20:49:28.939925 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.939872 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" event={"ID":"4486f93b-6513-4998-a2c7-d55f05805ee1","Type":"ContainerDied","Data":"c515302eae4abe714b256e759efc94e3673c95191dda76bc9c726d1253bd74ff"} Nov 02 20:49:28.939986 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.939930 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j" Nov 02 20:49:28.943549 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.943533 4971 scope.go:122] "RemoveContainer" containerID="1b44310cacce629cddf37549ff26e06391314064c0e9425a6f435f1b5b9e6d0d" Nov 02 20:49:28.943796 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:49:28.943775 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b44310cacce629cddf37549ff26e06391314064c0e9425a6f435f1b5b9e6d0d\": container with ID starting with 1b44310cacce629cddf37549ff26e06391314064c0e9425a6f435f1b5b9e6d0d not found: ID does not exist" containerID="1b44310cacce629cddf37549ff26e06391314064c0e9425a6f435f1b5b9e6d0d" Nov 02 20:49:28.943869 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.943808 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b44310cacce629cddf37549ff26e06391314064c0e9425a6f435f1b5b9e6d0d"} err="failed to get container status \"1b44310cacce629cddf37549ff26e06391314064c0e9425a6f435f1b5b9e6d0d\": rpc error: code = NotFound desc = could not find container \"1b44310cacce629cddf37549ff26e06391314064c0e9425a6f435f1b5b9e6d0d\": container with ID starting with 1b44310cacce629cddf37549ff26e06391314064c0e9425a6f435f1b5b9e6d0d not found: ID does not exist" Nov 02 20:49:28.943869 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.943832 4971 scope.go:122] "RemoveContainer" containerID="d2607e7731514643c5be9110331e89d06781e9f7f4595bde90be8e0d7307a008" Nov 02 20:49:28.954487 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.954462 4971 scope.go:122] "RemoveContainer" containerID="d2607e7731514643c5be9110331e89d06781e9f7f4595bde90be8e0d7307a008" Nov 02 20:49:28.954798 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:49:28.954779 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2607e7731514643c5be9110331e89d06781e9f7f4595bde90be8e0d7307a008\": container with ID starting with d2607e7731514643c5be9110331e89d06781e9f7f4595bde90be8e0d7307a008 not found: ID does not exist" containerID="d2607e7731514643c5be9110331e89d06781e9f7f4595bde90be8e0d7307a008" Nov 02 20:49:28.954862 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.954806 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2607e7731514643c5be9110331e89d06781e9f7f4595bde90be8e0d7307a008"} err="failed to get container status \"d2607e7731514643c5be9110331e89d06781e9f7f4595bde90be8e0d7307a008\": rpc error: code = NotFound desc = could not find container \"d2607e7731514643c5be9110331e89d06781e9f7f4595bde90be8e0d7307a008\": container with ID starting with d2607e7731514643c5be9110331e89d06781e9f7f4595bde90be8e0d7307a008 not found: ID does not exist" Nov 02 20:49:28.954862 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.954821 4971 scope.go:122] "RemoveContainer" containerID="cde583b432e41744fe7e87c7dfbd2f4ee8c394fe5b25f89d4e0e5b16d6026534" Nov 02 20:49:28.966017 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.965995 4971 scope.go:122] "RemoveContainer" containerID="cde583b432e41744fe7e87c7dfbd2f4ee8c394fe5b25f89d4e0e5b16d6026534" Nov 02 20:49:28.966360 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:49:28.966341 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde583b432e41744fe7e87c7dfbd2f4ee8c394fe5b25f89d4e0e5b16d6026534\": container with ID starting with cde583b432e41744fe7e87c7dfbd2f4ee8c394fe5b25f89d4e0e5b16d6026534 not found: ID does not exist" containerID="cde583b432e41744fe7e87c7dfbd2f4ee8c394fe5b25f89d4e0e5b16d6026534" Nov 02 20:49:28.966441 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:28.966368 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde583b432e41744fe7e87c7dfbd2f4ee8c394fe5b25f89d4e0e5b16d6026534"} err="failed to get container status \"cde583b432e41744fe7e87c7dfbd2f4ee8c394fe5b25f89d4e0e5b16d6026534\": rpc error: code = NotFound desc = could not find container \"cde583b432e41744fe7e87c7dfbd2f4ee8c394fe5b25f89d4e0e5b16d6026534\": container with ID starting with cde583b432e41744fe7e87c7dfbd2f4ee8c394fe5b25f89d4e0e5b16d6026534 not found: ID does not exist" Nov 02 20:49:29.086251 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:29.086218 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j"] Nov 02 20:49:29.090873 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:29.090851 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/premium-w3-61cdc1-kserve-5f6757cdcc-jww2j"] Nov 02 20:49:29.195427 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:29.195311 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s"] Nov 02 20:49:29.199134 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:29.199112 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/distinct-w3-61cdc1-kserve-864f7bcc76-nxx4s"] Nov 02 20:49:29.302494 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:29.302469 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t"] Nov 02 20:49:29.308784 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:29.308750 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/simulator-w3-61cdc1-kserve-955c8b8-gdb8t"] Nov 02 20:49:29.616548 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:29.616508 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" podUID="ee36cd67-b157-4fda-bd62-34ba6cd9460e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:49:29.616728 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:29.616611 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" podUID="e8daff05-2927-4e7b-95a4-201024607e67" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:49:29.866019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:29.865982 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" path="/var/lib/kubelet/pods/0cfa705d-47bf-4d68-8f92-a9132c911b6b/volumes" Nov 02 20:49:29.866750 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:29.866694 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" path="/var/lib/kubelet/pods/4486f93b-6513-4998-a2c7-d55f05805ee1/volumes" Nov 02 20:49:29.867269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:29.867251 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" path="/var/lib/kubelet/pods/aa6f0ad6-43f0-484a-9f00-55f7dcef5698/volumes" Nov 02 20:49:30.360155 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:30.360071 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:30.969885 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:30.969638 4971 generic.go:356] "Generic (PLEG): container finished" podID="ee36cd67-b157-4fda-bd62-34ba6cd9460e" containerID="b99d29e8e91d614e4104aab2c219fe72b407dd2273f525cd20a6e07546143efd" exitCode=0 Nov 02 20:49:30.969885 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:30.969815 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" event={"ID":"ee36cd67-b157-4fda-bd62-34ba6cd9460e","Type":"ContainerDied","Data":"b99d29e8e91d614e4104aab2c219fe72b407dd2273f525cd20a6e07546143efd"} Nov 02 20:49:30.973570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:30.973463 4971 generic.go:356] "Generic (PLEG): container finished" podID="e8daff05-2927-4e7b-95a4-201024607e67" containerID="0cd4f0a112f288becc144823eea65127280484fbb051bbd11ca1e8ec0aaa5b28" exitCode=0 Nov 02 20:49:30.973570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:30.973518 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" event={"ID":"e8daff05-2927-4e7b-95a4-201024607e67","Type":"ContainerDied","Data":"0cd4f0a112f288becc144823eea65127280484fbb051bbd11ca1e8ec0aaa5b28"} Nov 02 20:49:31.022576 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.017518 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" Nov 02 20:49:31.099107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.098431 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee36cd67-b157-4fda-bd62-34ba6cd9460e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee36cd67-b157-4fda-bd62-34ba6cd9460e-plugins-config-volume\") pod \"ee36cd67-b157-4fda-bd62-34ba6cd9460e\" (UID: \"ee36cd67-b157-4fda-bd62-34ba6cd9460e\") " Nov 02 20:49:31.099107 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.098510 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee36cd67-b157-4fda-bd62-34ba6cd9460e-kube-api-access-t5pqf\" (UniqueName: \"kubernetes.io/projected/ee36cd67-b157-4fda-bd62-34ba6cd9460e-kube-api-access-t5pqf\") pod \"ee36cd67-b157-4fda-bd62-34ba6cd9460e\" (UID: \"ee36cd67-b157-4fda-bd62-34ba6cd9460e\") " Nov 02 20:49:31.106003 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.099563 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee36cd67-b157-4fda-bd62-34ba6cd9460e-plugins-config-volume" pod "ee36cd67-b157-4fda-bd62-34ba6cd9460e" (UID: "ee36cd67-b157-4fda-bd62-34ba6cd9460e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:31.111112 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.109267 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee36cd67-b157-4fda-bd62-34ba6cd9460e-kube-api-access-t5pqf" pod "ee36cd67-b157-4fda-bd62-34ba6cd9460e" (UID: "ee36cd67-b157-4fda-bd62-34ba6cd9460e"). InnerVolumeSpecName "kube-api-access-t5pqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:31.200469 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.200282 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee36cd67-b157-4fda-bd62-34ba6cd9460e-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:31.200469 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.200312 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5pqf\" (UniqueName: \"kubernetes.io/projected/ee36cd67-b157-4fda-bd62-34ba6cd9460e-kube-api-access-t5pqf\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:31.286176 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.286063 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" Nov 02 20:49:31.402622 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.402586 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8daff05-2927-4e7b-95a4-201024607e67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8daff05-2927-4e7b-95a4-201024607e67-plugins-config-volume\") pod \"e8daff05-2927-4e7b-95a4-201024607e67\" (UID: \"e8daff05-2927-4e7b-95a4-201024607e67\") " Nov 02 20:49:31.403118 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.402709 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8daff05-2927-4e7b-95a4-201024607e67-kube-api-access-t99kx\" (UniqueName: \"kubernetes.io/projected/e8daff05-2927-4e7b-95a4-201024607e67-kube-api-access-t99kx\") pod \"e8daff05-2927-4e7b-95a4-201024607e67\" (UID: \"e8daff05-2927-4e7b-95a4-201024607e67\") " Nov 02 20:49:31.403693 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.403660 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8daff05-2927-4e7b-95a4-201024607e67-plugins-config-volume" pod "e8daff05-2927-4e7b-95a4-201024607e67" (UID: "e8daff05-2927-4e7b-95a4-201024607e67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:31.405549 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.405520 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8daff05-2927-4e7b-95a4-201024607e67-kube-api-access-t99kx" pod "e8daff05-2927-4e7b-95a4-201024607e67" (UID: "e8daff05-2927-4e7b-95a4-201024607e67"). InnerVolumeSpecName "kube-api-access-t99kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:31.503522 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.503437 4971 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8daff05-2927-4e7b-95a4-201024607e67-plugins-config-volume\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:31.503522 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.503467 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t99kx\" (UniqueName: \"kubernetes.io/projected/e8daff05-2927-4e7b-95a4-201024607e67-kube-api-access-t99kx\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:31.831422 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.831344 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:31.981190 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.981161 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" Nov 02 20:49:31.981390 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.981188 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn" event={"ID":"e8daff05-2927-4e7b-95a4-201024607e67","Type":"ContainerDied","Data":"82aa14b06550625f7e006146031342f2d72ef1edb4370b28fc69b819adb61e23"} Nov 02 20:49:31.981390 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.981254 4971 scope.go:122] "RemoveContainer" containerID="0cd4f0a112f288becc144823eea65127280484fbb051bbd11ca1e8ec0aaa5b28" Nov 02 20:49:31.984450 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.984422 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" event={"ID":"ee36cd67-b157-4fda-bd62-34ba6cd9460e","Type":"ContainerDied","Data":"3200eb5efc6a00ee9ec0c1e591a648c2072aae70f10c250866c07b55c6e91a91"} Nov 02 20:49:31.984623 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.984605 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj" Nov 02 20:49:31.992979 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:31.992958 4971 scope.go:122] "RemoveContainer" containerID="b99d29e8e91d614e4104aab2c219fe72b407dd2273f525cd20a6e07546143efd" Nov 02 20:49:32.137993 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:32.137960 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn"] Nov 02 20:49:32.185304 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:32.185273 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a5cad4-b8445d88c-ljfmn"] Nov 02 20:49:32.222051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:32.222023 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:49:32.224274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:32.224245 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr"] Nov 02 20:49:32.228918 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:32.228882 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a5cad4-j9gmr"] Nov 02 20:49:32.229770 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:32.229749 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:49:32.244093 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:32.244056 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj"] Nov 02 20:49:32.250403 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:32.250381 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a5cad4-6b6c797f78-k2wkj"] Nov 02 20:49:33.342480 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:49:33.342443 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:49:33.565146 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:33.565110 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:33.865537 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:33.865506 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c85642-5d6f-49de-8ba4-5cd2a9f543d0" path="/var/lib/kubelet/pods/c7c85642-5d6f-49de-8ba4-5cd2a9f543d0/volumes" Nov 02 20:49:33.865875 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:33.865861 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8daff05-2927-4e7b-95a4-201024607e67" path="/var/lib/kubelet/pods/e8daff05-2927-4e7b-95a4-201024607e67/volumes" Nov 02 20:49:33.866234 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:33.866220 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee36cd67-b157-4fda-bd62-34ba6cd9460e" path="/var/lib/kubelet/pods/ee36cd67-b157-4fda-bd62-34ba6cd9460e/volumes" Nov 02 20:49:33.905339 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:33.905311 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:35.978896 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:35.978859 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:36.224071 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:36.224036 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:36.524792 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:36.524752 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:39.050327 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:39.050291 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:39.277405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:39.277375 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9"] Nov 02 20:49:39.277603 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:39.277582 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" podUID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" containerName="istio-proxy" containerID="cri-o://4f37038f58f86692833fddc244ef59f7a885c316c8a8526af2065da734fda6be" gracePeriod=30 Nov 02 20:49:39.304982 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:39.304921 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:39.604365 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:39.604329 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:39.853917 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:39.853880 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:41.070443 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:41.070398 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:41.405878 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:41.405841 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:41.460425 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:41.460396 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p"] Nov 02 20:49:41.507325 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:41.507297 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-61cdc1-p8n8p"] Nov 02 20:49:41.865029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:41.864996 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b63506f-d59c-4b46-81d9-aa1f8968f8c0" path="/var/lib/kubelet/pods/4b63506f-d59c-4b46-81d9-aa1f8968f8c0/volumes" Nov 02 20:49:42.006930 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:42.006888 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:42.248682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:42.248609 4971 scope.go:122] "RemoveContainer" containerID="a5ceee3a2ba723d349a076a1657f527104553a487956fb34ee30b6a4ee94e18c" Nov 02 20:49:42.258438 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:42.258420 4971 scope.go:122] "RemoveContainer" containerID="7e1d99c5653a85d32887db7c57f038a99500bd3ac23f473dcae9a8f05d8cc150" Nov 02 20:49:42.378432 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:42.378402 4971 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:49:42.378564 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:42.378464 4971 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" podUID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:49:43.360688 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:43.360645 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:44.529117 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.529064 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:49:44.624356 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.624328 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-workload-socket\") pod \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " Nov 02 20:49:44.624500 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.624368 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-credential-socket\") pod \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " Nov 02 20:49:44.624500 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.624397 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istiod-ca-cert\") pod \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " Nov 02 20:49:44.624500 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.624443 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-workload-certs\") pod \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " Nov 02 20:49:44.624500 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.624475 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-envoy\") pod \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " Nov 02 20:49:44.624679 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.624545 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-token\" (UniqueName: \"kubernetes.io/projected/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-token\") pod \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " Nov 02 20:49:44.624679 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.624617 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41baf9c8-0dc6-4e97-958c-70391cbeb98b-kube-api-access-x2nsp\" (UniqueName: \"kubernetes.io/projected/41baf9c8-0dc6-4e97-958c-70391cbeb98b-kube-api-access-x2nsp\") pod \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " Nov 02 20:49:44.624679 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.624664 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-data\") pod \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " Nov 02 20:49:44.624844 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.624615 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-workload-socket" pod "41baf9c8-0dc6-4e97-958c-70391cbeb98b" (UID: "41baf9c8-0dc6-4e97-958c-70391cbeb98b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:44.624844 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.624711 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-podinfo\") pod \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\" (UID: \"41baf9c8-0dc6-4e97-958c-70391cbeb98b\") " Nov 02 20:49:44.624844 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.624669 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-credential-socket" pod "41baf9c8-0dc6-4e97-958c-70391cbeb98b" (UID: "41baf9c8-0dc6-4e97-958c-70391cbeb98b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:44.624844 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.624760 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-workload-certs" pod "41baf9c8-0dc6-4e97-958c-70391cbeb98b" (UID: "41baf9c8-0dc6-4e97-958c-70391cbeb98b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:44.625072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.624893 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istiod-ca-cert" pod "41baf9c8-0dc6-4e97-958c-70391cbeb98b" (UID: "41baf9c8-0dc6-4e97-958c-70391cbeb98b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:44.625072 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.625031 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-data" pod "41baf9c8-0dc6-4e97-958c-70391cbeb98b" (UID: "41baf9c8-0dc6-4e97-958c-70391cbeb98b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:44.625570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.625307 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:44.625570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.625491 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:44.625570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.625510 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:44.625570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.625523 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:44.625570 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.625538 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:44.626970 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.626939 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-envoy" pod "41baf9c8-0dc6-4e97-958c-70391cbeb98b" (UID: "41baf9c8-0dc6-4e97-958c-70391cbeb98b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:44.627127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.627106 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41baf9c8-0dc6-4e97-958c-70391cbeb98b-kube-api-access-x2nsp" pod "41baf9c8-0dc6-4e97-958c-70391cbeb98b" (UID: "41baf9c8-0dc6-4e97-958c-70391cbeb98b"). InnerVolumeSpecName "kube-api-access-x2nsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:44.627490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.627467 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-token" pod "41baf9c8-0dc6-4e97-958c-70391cbeb98b" (UID: "41baf9c8-0dc6-4e97-958c-70391cbeb98b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:44.627490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.627470 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-podinfo" pod "41baf9c8-0dc6-4e97-958c-70391cbeb98b" (UID: "41baf9c8-0dc6-4e97-958c-70391cbeb98b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:49:44.727014 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.726990 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:44.727014 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.727013 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:44.727202 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.727025 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2nsp\" (UniqueName: \"kubernetes.io/projected/41baf9c8-0dc6-4e97-958c-70391cbeb98b-kube-api-access-x2nsp\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:44.727202 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:44.727034 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41baf9c8-0dc6-4e97-958c-70391cbeb98b-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:45.074340 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:45.074252 4971 generic.go:356] "Generic (PLEG): container finished" podID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" containerID="4f37038f58f86692833fddc244ef59f7a885c316c8a8526af2065da734fda6be" exitCode=0 Nov 02 20:49:45.074340 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:45.074299 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" Nov 02 20:49:45.074340 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:45.074328 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" event={"ID":"41baf9c8-0dc6-4e97-958c-70391cbeb98b","Type":"ContainerDied","Data":"4f37038f58f86692833fddc244ef59f7a885c316c8a8526af2065da734fda6be"} Nov 02 20:49:45.074648 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:45.074379 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9" event={"ID":"41baf9c8-0dc6-4e97-958c-70391cbeb98b","Type":"ContainerDied","Data":"803ef52e54a2fc3bb1abbbb9fa20d9898bf9068b488dd5d1da88301ba133fc76"} Nov 02 20:49:45.074648 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:45.074401 4971 scope.go:122] "RemoveContainer" containerID="4f37038f58f86692833fddc244ef59f7a885c316c8a8526af2065da734fda6be" Nov 02 20:49:45.084863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:45.084847 4971 scope.go:122] "RemoveContainer" containerID="ce529c7e4d15bf92455208c281ff7dcf850d819a17919501a8153c17fc9ab454" Nov 02 20:49:45.093563 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:45.093544 4971 scope.go:122] "RemoveContainer" containerID="4f37038f58f86692833fddc244ef59f7a885c316c8a8526af2065da734fda6be" Nov 02 20:49:45.093823 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:49:45.093805 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f37038f58f86692833fddc244ef59f7a885c316c8a8526af2065da734fda6be\": container with ID starting with 4f37038f58f86692833fddc244ef59f7a885c316c8a8526af2065da734fda6be not found: ID does not exist" containerID="4f37038f58f86692833fddc244ef59f7a885c316c8a8526af2065da734fda6be" Nov 02 20:49:45.093901 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:45.093836 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f37038f58f86692833fddc244ef59f7a885c316c8a8526af2065da734fda6be"} err="failed to get container status \"4f37038f58f86692833fddc244ef59f7a885c316c8a8526af2065da734fda6be\": rpc error: code = NotFound desc = could not find container \"4f37038f58f86692833fddc244ef59f7a885c316c8a8526af2065da734fda6be\": container with ID starting with 4f37038f58f86692833fddc244ef59f7a885c316c8a8526af2065da734fda6be not found: ID does not exist" Nov 02 20:49:45.093901 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:45.093859 4971 scope.go:122] "RemoveContainer" containerID="ce529c7e4d15bf92455208c281ff7dcf850d819a17919501a8153c17fc9ab454" Nov 02 20:49:45.094110 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:49:45.094074 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce529c7e4d15bf92455208c281ff7dcf850d819a17919501a8153c17fc9ab454\": container with ID starting with ce529c7e4d15bf92455208c281ff7dcf850d819a17919501a8153c17fc9ab454 not found: ID does not exist" containerID="ce529c7e4d15bf92455208c281ff7dcf850d819a17919501a8153c17fc9ab454" Nov 02 20:49:45.094158 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:45.094117 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce529c7e4d15bf92455208c281ff7dcf850d819a17919501a8153c17fc9ab454"} err="failed to get container status \"ce529c7e4d15bf92455208c281ff7dcf850d819a17919501a8153c17fc9ab454\": rpc error: code = NotFound desc = could not find container \"ce529c7e4d15bf92455208c281ff7dcf850d819a17919501a8153c17fc9ab454\": container with ID starting with ce529c7e4d15bf92455208c281ff7dcf850d819a17919501a8153c17fc9ab454 not found: ID does not exist" Nov 02 20:49:45.230744 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:45.230718 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9"] Nov 02 20:49:45.277519 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:45.277497 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b5wq9"] Nov 02 20:49:45.865685 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:45.865651 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" path="/var/lib/kubelet/pods/41baf9c8-0dc6-4e97-958c-70391cbeb98b/volumes" Nov 02 20:49:47.823453 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:47.823423 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:48.093837 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:48.093756 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:48.339237 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:48.339181 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:50.406816 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:50.406781 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:50.653793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:50.653759 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:50.906027 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:50.905992 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:53.885907 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:53.885870 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:56.819483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:56.819449 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:59.003365 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:59.003330 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:49:59.252953 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:49:59.252919 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:50:03.341573 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:50:03.341529 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:50:05.917418 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917373 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-569554b5f8-8pdkd"] Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917811 4971 cpu_manager.go:414] "containerMap: removing container" podUID="ee36cd67-b157-4fda-bd62-34ba6cd9460e" containerName="payload-processing" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917824 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee36cd67-b157-4fda-bd62-34ba6cd9460e" containerName="payload-processing" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917837 4971 cpu_manager.go:414] "containerMap: removing container" podUID="b46adb19-9afd-4671-94fe-4a5f6b965dfb" containerName="revoke-keys" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917842 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46adb19-9afd-4671-94fe-4a5f6b965dfb" containerName="revoke-keys" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917855 4971 cpu_manager.go:414] "containerMap: removing container" podUID="e8daff05-2927-4e7b-95a4-201024607e67" containerName="payload-pre-processing" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917860 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8daff05-2927-4e7b-95a4-201024607e67" containerName="payload-pre-processing" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917870 4971 cpu_manager.go:414] "containerMap: removing container" podUID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" containerName="istio-proxy" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917874 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" containerName="istio-proxy" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917888 4971 cpu_manager.go:414] "containerMap: removing container" podUID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" containerName="istio-proxy" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917892 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" containerName="istio-proxy" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917898 4971 cpu_manager.go:414] "containerMap: removing container" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917903 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917910 4971 cpu_manager.go:414] "containerMap: removing container" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917914 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917921 4971 cpu_manager.go:414] "containerMap: removing container" podUID="e88e7eef-5eb8-4274-ad74-4c691a426260" containerName="revoke-keys" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917926 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e88e7eef-5eb8-4274-ad74-4c691a426260" containerName="revoke-keys" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917934 4971 cpu_manager.go:414] "containerMap: removing container" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" Nov 02 20:50:05.917942 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.917939 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" Nov 02 20:50:05.919239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.918012 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee36cd67-b157-4fda-bd62-34ba6cd9460e" containerName="payload-processing" Nov 02 20:50:05.919239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.918022 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46adb19-9afd-4671-94fe-4a5f6b965dfb" containerName="revoke-keys" Nov 02 20:50:05.919239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.918030 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8daff05-2927-4e7b-95a4-201024607e67" containerName="payload-pre-processing" Nov 02 20:50:05.919239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.918036 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" containerName="istio-proxy" Nov 02 20:50:05.919239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.918043 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="41baf9c8-0dc6-4e97-958c-70391cbeb98b" containerName="istio-proxy" Nov 02 20:50:05.919239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.918048 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa6f0ad6-43f0-484a-9f00-55f7dcef5698" containerName="main" Nov 02 20:50:05.919239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.918054 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cfa705d-47bf-4d68-8f92-a9132c911b6b" containerName="main" Nov 02 20:50:05.919239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.918059 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="e88e7eef-5eb8-4274-ad74-4c691a426260" containerName="revoke-keys" Nov 02 20:50:05.919239 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.918065 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="4486f93b-6513-4998-a2c7-d55f05805ee1" containerName="main" Nov 02 20:50:05.921428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:05.921407 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569554b5f8-8pdkd" Nov 02 20:50:06.035115 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.034207 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-569554b5f8-8pdkd"] Nov 02 20:50:06.035115 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.034409 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-r6lrl\"" Nov 02 20:50:06.035115 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.035117 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 20:50:06.035431 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.035197 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 20:50:06.050897 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.050862 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 20:50:06.137727 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.137692 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de714a26-2a3e-4256-a433-d6c9afc4161c-webhook-cert\") pod \"maas-controller-569554b5f8-8pdkd\" (UID: \"de714a26-2a3e-4256-a433-d6c9afc4161c\") " pod="opendatahub/maas-controller-569554b5f8-8pdkd" Nov 02 20:50:06.137899 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.137762 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de714a26-2a3e-4256-a433-d6c9afc4161c-metrics-certs\") pod \"maas-controller-569554b5f8-8pdkd\" (UID: \"de714a26-2a3e-4256-a433-d6c9afc4161c\") " pod="opendatahub/maas-controller-569554b5f8-8pdkd" Nov 02 20:50:06.137899 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.137866 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlgcr\" (UniqueName: \"kubernetes.io/projected/de714a26-2a3e-4256-a433-d6c9afc4161c-kube-api-access-hlgcr\") pod \"maas-controller-569554b5f8-8pdkd\" (UID: \"de714a26-2a3e-4256-a433-d6c9afc4161c\") " pod="opendatahub/maas-controller-569554b5f8-8pdkd" Nov 02 20:50:06.239174 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.239072 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de714a26-2a3e-4256-a433-d6c9afc4161c-webhook-cert\") pod \"maas-controller-569554b5f8-8pdkd\" (UID: \"de714a26-2a3e-4256-a433-d6c9afc4161c\") " pod="opendatahub/maas-controller-569554b5f8-8pdkd" Nov 02 20:50:06.239362 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.239183 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de714a26-2a3e-4256-a433-d6c9afc4161c-metrics-certs\") pod \"maas-controller-569554b5f8-8pdkd\" (UID: \"de714a26-2a3e-4256-a433-d6c9afc4161c\") " pod="opendatahub/maas-controller-569554b5f8-8pdkd" Nov 02 20:50:06.239362 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.239222 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlgcr\" (UniqueName: \"kubernetes.io/projected/de714a26-2a3e-4256-a433-d6c9afc4161c-kube-api-access-hlgcr\") pod \"maas-controller-569554b5f8-8pdkd\" (UID: \"de714a26-2a3e-4256-a433-d6c9afc4161c\") " pod="opendatahub/maas-controller-569554b5f8-8pdkd" Nov 02 20:50:06.241812 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.241779 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de714a26-2a3e-4256-a433-d6c9afc4161c-webhook-cert\") pod \"maas-controller-569554b5f8-8pdkd\" (UID: \"de714a26-2a3e-4256-a433-d6c9afc4161c\") " pod="opendatahub/maas-controller-569554b5f8-8pdkd" Nov 02 20:50:06.241940 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.241817 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de714a26-2a3e-4256-a433-d6c9afc4161c-metrics-certs\") pod \"maas-controller-569554b5f8-8pdkd\" (UID: \"de714a26-2a3e-4256-a433-d6c9afc4161c\") " pod="opendatahub/maas-controller-569554b5f8-8pdkd" Nov 02 20:50:06.297244 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.297212 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlgcr\" (UniqueName: \"kubernetes.io/projected/de714a26-2a3e-4256-a433-d6c9afc4161c-kube-api-access-hlgcr\") pod \"maas-controller-569554b5f8-8pdkd\" (UID: \"de714a26-2a3e-4256-a433-d6c9afc4161c\") " pod="opendatahub/maas-controller-569554b5f8-8pdkd" Nov 02 20:50:06.553045 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.552966 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569554b5f8-8pdkd" Nov 02 20:50:06.925863 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.925830 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-569554b5f8-8pdkd"] Nov 02 20:50:06.926828 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:50:06.926797 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde714a26_2a3e_4256_a433_d6c9afc4161c.slice/crio-63438832e7997ce3cdc588ee1a2617f988f54ba76744a804e90364478df5beef WatchSource:0}: Error finding container 63438832e7997ce3cdc588ee1a2617f988f54ba76744a804e90364478df5beef: Status 404 returned error can't find the container with id 63438832e7997ce3cdc588ee1a2617f988f54ba76744a804e90364478df5beef Nov 02 20:50:06.928462 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:06.928444 4971 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:50:07.212654 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:07.212572 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569554b5f8-8pdkd" event={"ID":"de714a26-2a3e-4256-a433-d6c9afc4161c","Type":"ContainerStarted","Data":"63438832e7997ce3cdc588ee1a2617f988f54ba76744a804e90364478df5beef"} Nov 02 20:50:08.772848 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:08.772809 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:50:09.019332 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:09.019287 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:50:11.246871 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:11.246828 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569554b5f8-8pdkd" event={"ID":"de714a26-2a3e-4256-a433-d6c9afc4161c","Type":"ContainerStarted","Data":"c9b062c8935c87c696a35d1ab7caba454da829df8dd19549ebfade073079ff86"} Nov 02 20:50:11.247324 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:11.246923 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-569554b5f8-8pdkd" Nov 02 20:50:11.361408 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:11.361351 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-569554b5f8-8pdkd" podStartSLOduration=2.752330943 podStartE2EDuration="6.361333065s" totalImagesPullingTime="3.609002122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:50:11.360959835 +0000 UTC m=+2139.650616364" watchObservedRunningTime="2026-11-02 20:50:11.361333065 +0000 UTC m=+2139.650989594" Nov 02 20:50:22.056045 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:22.056004 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7"] Nov 02 20:50:22.102399 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:22.102369 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434192b6-qknx7"] Nov 02 20:50:22.255619 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:22.255595 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-569554b5f8-8pdkd" Nov 02 20:50:23.056763 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:23.056730 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv"] Nov 02 20:50:23.103579 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:23.103551 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8719adfc-qv6nv"] Nov 02 20:50:23.865676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:23.865643 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63eaa68e-23a6-4d8d-ad62-5e795706b24d" path="/var/lib/kubelet/pods/63eaa68e-23a6-4d8d-ad62-5e795706b24d/volumes" Nov 02 20:50:23.866042 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:23.866029 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe71e0b5-fe81-4fff-963a-ea781eaceb14" path="/var/lib/kubelet/pods/fe71e0b5-fe81-4fff-963a-ea781eaceb14/volumes" Nov 02 20:50:26.444577 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.444541 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx"] Nov 02 20:50:26.448458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.448437 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.491669 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.491645 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm"] Nov 02 20:50:26.495248 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.495232 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.541935 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.541908 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.542067 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.541953 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.542067 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.541988 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.542067 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.542044 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.542300 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.542236 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.542300 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.542273 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.542380 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.542354 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.542417 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.542387 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.542457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.542433 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.542518 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.542491 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2sfb\" (UniqueName: \"kubernetes.io/projected/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-kube-api-access-x2sfb\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.542565 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.542528 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.542612 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.542591 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2plr\" (UniqueName: \"kubernetes.io/projected/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-kube-api-access-k2plr\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.542654 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.542625 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.542698 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.542671 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.542698 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.542687 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.542807 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.542711 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.542867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.542809 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.542867 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.542845 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.552202 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.552176 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx"] Nov 02 20:50:26.560432 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.560407 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm"] Nov 02 20:50:26.643392 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643365 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.643543 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643401 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.643543 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643426 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.643543 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643479 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.643543 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643521 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.643799 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643547 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.643799 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643573 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.643799 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643607 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.643799 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643644 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.643799 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643669 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.643799 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643755 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.643799 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643782 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.644122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643830 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.644122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643846 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.644122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643860 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.644122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643888 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.644122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643930 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.644122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643949 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.644122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.643967 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2sfb\" (UniqueName: \"kubernetes.io/projected/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-kube-api-access-x2sfb\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.644122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.644018 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.644122 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.644053 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2plr\" (UniqueName: \"kubernetes.io/projected/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-kube-api-access-k2plr\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.644466 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.644212 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.644466 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.644212 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.644466 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.644262 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.644632 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.644490 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.644719 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.644689 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.644945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.644693 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.644945 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.644728 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.646586 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.646558 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.646793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.646774 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.646887 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.646871 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.646969 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.646947 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.701643 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.701589 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2plr\" (UniqueName: \"kubernetes.io/projected/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-kube-api-access-k2plr\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.701755 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.701665 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.701814 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.701757 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-t9vhx\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.701814 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.701792 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2sfb\" (UniqueName: \"kubernetes.io/projected/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-kube-api-access-x2sfb\") pod \"maas-default-gateway-openshift-default-85544d4cd7-p96bm\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:26.761368 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.761332 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:26.807537 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:26.807503 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:27.178855 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:27.178829 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx"] Nov 02 20:50:27.180114 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:50:27.180068 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb306a7c_bde9_43de_9b5a_bf0cc76a6ec8.slice/crio-ed6283828ec6f35e82eba53f9b564e6ed0ff118584336a4963b2f261e341984e WatchSource:0}: Error finding container ed6283828ec6f35e82eba53f9b564e6ed0ff118584336a4963b2f261e341984e: Status 404 returned error can't find the container with id ed6283828ec6f35e82eba53f9b564e6ed0ff118584336a4963b2f261e341984e Nov 02 20:50:27.182386 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:27.182348 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:50:27.182511 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:27.182426 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:50:27.182511 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:27.182469 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:50:27.228221 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:27.228200 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm"] Nov 02 20:50:27.229524 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:50:27.229490 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c4c5a9_6da8_4d19_bad1_7fd9c8291349.slice/crio-0992be9d5bab4fd57a9767904496ddb40fd469282726671bb281b25ab43e6d10 WatchSource:0}: Error finding container 0992be9d5bab4fd57a9767904496ddb40fd469282726671bb281b25ab43e6d10: Status 404 returned error can't find the container with id 0992be9d5bab4fd57a9767904496ddb40fd469282726671bb281b25ab43e6d10 Nov 02 20:50:27.231858 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:27.231825 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:50:27.231951 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:27.231891 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:50:27.231951 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:27.231918 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:50:27.352071 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:27.352033 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" event={"ID":"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8","Type":"ContainerStarted","Data":"ba93a618221c6aad8520676c9bbd8018b76b814151a2899698787626a757a537"} Nov 02 20:50:27.352286 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:27.352111 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" event={"ID":"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8","Type":"ContainerStarted","Data":"ed6283828ec6f35e82eba53f9b564e6ed0ff118584336a4963b2f261e341984e"} Nov 02 20:50:27.353970 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:27.353941 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" event={"ID":"69c4c5a9-6da8-4d19-bad1-7fd9c8291349","Type":"ContainerStarted","Data":"199608e47abe389bdf2ebe26f2bc37519c4e1d893a7d261534a84fc129070747"} Nov 02 20:50:27.354135 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:27.353986 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" event={"ID":"69c4c5a9-6da8-4d19-bad1-7fd9c8291349","Type":"ContainerStarted","Data":"0992be9d5bab4fd57a9767904496ddb40fd469282726671bb281b25ab43e6d10"} Nov 02 20:50:27.468188 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:27.468026 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" podStartSLOduration=1.468012938 podStartE2EDuration="1.468012938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:50:27.465554144 +0000 UTC m=+2155.755210709" watchObservedRunningTime="2026-11-02 20:50:27.468012938 +0000 UTC m=+2155.757669467" Nov 02 20:50:27.583459 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:27.583398 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" podStartSLOduration=1.583376779 podStartE2EDuration="1.583376779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:50:27.578900448 +0000 UTC m=+2155.868557014" watchObservedRunningTime="2026-11-02 20:50:27.583376779 +0000 UTC m=+2155.873033307" Nov 02 20:50:27.764167 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:27.763795 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:27.807923 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:27.807893 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:28.766224 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:28.766190 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-t9vhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Nov 02 20:50:28.766793 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:28.766765 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" podUID="eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Nov 02 20:50:28.809657 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:28.809635 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-p96bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 02 20:50:28.809845 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:28.809824 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" podUID="69c4c5a9-6da8-4d19-bad1-7fd9c8291349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 02 20:50:29.764512 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:29.764469 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-t9vhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Nov 02 20:50:29.764833 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:29.764802 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" podUID="eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Nov 02 20:50:29.808011 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:29.807977 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-p96bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 02 20:50:29.808435 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:29.808026 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" podUID="69c4c5a9-6da8-4d19-bad1-7fd9c8291349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 02 20:50:30.358606 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.358562 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx"] Nov 02 20:50:30.376126 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.375371 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx" Nov 02 20:50:30.416259 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.416232 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx"] Nov 02 20:50:30.474270 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.474236 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:50:30.474659 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.474636 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:50:30.502528 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.502500 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-d6dsx\" (UID: \"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx" Nov 02 20:50:30.502691 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.502669 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9qrg\" (UniqueName: \"kubernetes.io/projected/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-kube-api-access-w9qrg\") pod \"maas-api-revoke-sub-85237e45-d6dsx\" (UID: \"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx" Nov 02 20:50:30.502834 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.502814 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-d6dsx\" (UID: \"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx" Nov 02 20:50:30.603565 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.603524 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-d6dsx\" (UID: \"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx" Nov 02 20:50:30.603774 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.603604 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9qrg\" (UniqueName: \"kubernetes.io/projected/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-kube-api-access-w9qrg\") pod \"maas-api-revoke-sub-85237e45-d6dsx\" (UID: \"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx" Nov 02 20:50:30.603774 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.603650 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-d6dsx\" (UID: \"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx" Nov 02 20:50:30.604335 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.604309 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-d6dsx\" (UID: \"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx" Nov 02 20:50:30.606013 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.605989 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-d6dsx\" (UID: \"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx" Nov 02 20:50:30.667847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.667770 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9qrg\" (UniqueName: \"kubernetes.io/projected/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-kube-api-access-w9qrg\") pod \"maas-api-revoke-sub-85237e45-d6dsx\" (UID: \"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx" Nov 02 20:50:30.694599 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.694565 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx" Nov 02 20:50:30.766111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.765483 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-t9vhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Nov 02 20:50:30.766111 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.765546 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" podUID="eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Nov 02 20:50:30.808663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.808376 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-p96bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 02 20:50:30.808663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:30.808451 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" podUID="69c4c5a9-6da8-4d19-bad1-7fd9c8291349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 02 20:50:31.119001 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:31.118974 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx"] Nov 02 20:50:31.121409 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:50:31.121378 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85e8926_09a1_4e2f_a110_0e6b17c7a3f2.slice/crio-6fdaf36c32dffc9b7e91e0fd98cc59fd686abe6fe96ce021c191872a1c356287 WatchSource:0}: Error finding container 6fdaf36c32dffc9b7e91e0fd98cc59fd686abe6fe96ce021c191872a1c356287: Status 404 returned error can't find the container with id 6fdaf36c32dffc9b7e91e0fd98cc59fd686abe6fe96ce021c191872a1c356287 Nov 02 20:50:31.400149 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:31.400048 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx" event={"ID":"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2","Type":"ContainerStarted","Data":"6fdaf36c32dffc9b7e91e0fd98cc59fd686abe6fe96ce021c191872a1c356287"} Nov 02 20:50:31.764662 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:31.764560 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-t9vhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Nov 02 20:50:31.764962 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:31.764913 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" podUID="eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Nov 02 20:50:31.807847 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:31.807817 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-p96bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 02 20:50:31.808019 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:31.808000 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" podUID="69c4c5a9-6da8-4d19-bad1-7fd9c8291349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 02 20:50:32.409169 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:32.409135 4971 generic.go:356] "Generic (PLEG): container finished" podID="c85e8926-09a1-4e2f-a110-0e6b17c7a3f2" containerID="3a467d114173b3408abd710d84163bea283e824744b705ee208fed2ec46a80c8" exitCode=0 Nov 02 20:50:32.409547 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:32.409218 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx" event={"ID":"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2","Type":"ContainerDied","Data":"3a467d114173b3408abd710d84163bea283e824744b705ee208fed2ec46a80c8"} Nov 02 20:50:32.765108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:32.764446 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-t9vhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Nov 02 20:50:32.765108 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:32.764510 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" podUID="eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Nov 02 20:50:32.808137 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:32.808103 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-p96bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 02 20:50:32.808301 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:32.808153 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" podUID="69c4c5a9-6da8-4d19-bad1-7fd9c8291349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 02 20:50:33.339562 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:50:33.339522 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:50:33.562611 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:33.562585 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx" Nov 02 20:50:33.669919 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:33.669895 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-kube-api-access-w9qrg\" (UniqueName: \"kubernetes.io/projected/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-kube-api-access-w9qrg\") pod \"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2\" (UID: \"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2\") " Nov 02 20:50:33.670070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:33.669975 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-maas-api-openshift-ca\") pod \"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2\" (UID: \"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2\") " Nov 02 20:50:33.670070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:33.670018 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-maas-api-xks-ca\") pod \"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2\" (UID: \"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2\") " Nov 02 20:50:33.670369 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:33.670342 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-maas-api-openshift-ca" pod "c85e8926-09a1-4e2f-a110-0e6b17c7a3f2" (UID: "c85e8926-09a1-4e2f-a110-0e6b17c7a3f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:50:33.670458 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:33.670441 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-maas-api-openshift-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:33.672569 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:33.672537 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-maas-api-xks-ca" pod "c85e8926-09a1-4e2f-a110-0e6b17c7a3f2" (UID: "c85e8926-09a1-4e2f-a110-0e6b17c7a3f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:50:33.672668 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:33.672572 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-kube-api-access-w9qrg" pod "c85e8926-09a1-4e2f-a110-0e6b17c7a3f2" (UID: "c85e8926-09a1-4e2f-a110-0e6b17c7a3f2"). InnerVolumeSpecName "kube-api-access-w9qrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:50:33.762298 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:33.762261 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-t9vhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Nov 02 20:50:33.762470 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:33.762327 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" podUID="eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Nov 02 20:50:33.771830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:33.771807 4971 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-maas-api-xks-ca\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:33.771830 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:33.771831 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9qrg\" (UniqueName: \"kubernetes.io/projected/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2-kube-api-access-w9qrg\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:33.807939 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:33.807913 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-p96bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Nov 02 20:50:33.808057 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:33.807988 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" podUID="69c4c5a9-6da8-4d19-bad1-7fd9c8291349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Nov 02 20:50:34.150058 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:34.150022 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:50:34.424314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:34.424229 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx" event={"ID":"c85e8926-09a1-4e2f-a110-0e6b17c7a3f2","Type":"ContainerDied","Data":"6fdaf36c32dffc9b7e91e0fd98cc59fd686abe6fe96ce021c191872a1c356287"} Nov 02 20:50:34.424314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:34.424254 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx" Nov 02 20:50:34.424606 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:34.424322 4971 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fdaf36c32dffc9b7e91e0fd98cc59fd686abe6fe96ce021c191872a1c356287" Nov 02 20:50:34.762504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:34.762418 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-t9vhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Nov 02 20:50:34.762504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:34.762488 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" podUID="eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Nov 02 20:50:35.808749 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:35.808708 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-p96bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:50:35.809230 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:35.808794 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" podUID="69c4c5a9-6da8-4d19-bad1-7fd9c8291349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:50:36.804227 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:36.804194 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-t9vhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:50:36.804429 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:36.804250 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" podUID="eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:50:36.850338 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:36.850236 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-p96bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:50:36.850338 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:36.850294 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" podUID="69c4c5a9-6da8-4d19-bad1-7fd9c8291349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:50:37.845254 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:37.845210 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-t9vhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:50:37.845464 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:37.845278 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" podUID="eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:50:37.850902 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:37.850872 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-p96bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:50:37.851391 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:37.850931 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" podUID="69c4c5a9-6da8-4d19-bad1-7fd9c8291349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:50:37.883101 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:37.883064 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:37.883453 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:37.883429 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:37.884132 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:37.884116 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:50:37.909667 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:37.909646 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:37.909931 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:37.909906 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:37.910609 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:37.910594 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:50:41.575566 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.575531 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w"] Nov 02 20:50:41.576195 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.576174 4971 cpu_manager.go:414] "containerMap: removing container" podUID="c85e8926-09a1-4e2f-a110-0e6b17c7a3f2" containerName="revoke-keys" Nov 02 20:50:41.576304 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.576195 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85e8926-09a1-4e2f-a110-0e6b17c7a3f2" containerName="revoke-keys" Nov 02 20:50:41.576366 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.576355 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85e8926-09a1-4e2f-a110-0e6b17c7a3f2" containerName="revoke-keys" Nov 02 20:50:41.580663 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.580643 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.676411 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.676382 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w"] Nov 02 20:50:41.751003 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.750972 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.751201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.751137 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.751201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.751170 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj2d5\" (UniqueName: \"kubernetes.io/projected/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-kube-api-access-zj2d5\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.751322 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.751203 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.751322 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.751310 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.751430 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.751347 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.751430 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.751365 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.751552 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.751446 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.751552 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.751530 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.852939 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.852911 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.853109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.852949 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.853109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.852966 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.853109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.852988 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.853109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.853039 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.853109 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.853059 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.853428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.853168 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.853428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.853238 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj2d5\" (UniqueName: \"kubernetes.io/projected/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-kube-api-access-zj2d5\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.853428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.853297 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.853428 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.853367 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.853676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.853445 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.853676 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.853503 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.853765 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.853710 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.853930 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.853913 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.855757 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.855725 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.855930 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.855913 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.910785 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.910764 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:41.910982 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:41.910962 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj2d5\" (UniqueName: \"kubernetes.io/projected/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-kube-api-access-zj2d5\") pod \"maas-default-gateway-openshift-default-85544d4cd7-rvs7w\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:42.194103 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:42.194003 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:42.335161 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:42.335124 4971 scope.go:122] "RemoveContainer" containerID="1c9fc42763cbb7cd9f24bc3b3899e3e5fedf0e90e0dd05f52385e7417d39ebdf" Nov 02 20:50:42.344743 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:42.344718 4971 scope.go:122] "RemoveContainer" containerID="8c024e201f338afbab80c8ecdf0ddc8eeb9c379a3189223937a09935d5bcc8d5" Nov 02 20:50:42.353641 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:42.353623 4971 scope.go:122] "RemoveContainer" containerID="55a21598be72020e2a01eb253514fc73387b33041d40050d35ea5d75faa2fa19" Nov 02 20:50:42.614970 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:42.614945 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w"] Nov 02 20:50:42.616228 ip-10-0-140-120 kubenswrapper[4971]: W1102 20:50:42.616199 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff2ae3c_c1ce_4db3_8ac7_1ee6ac803633.slice/crio-29482813d34a271ca6c661f440c81bcc73fa6740614e4946c78e3995a90f4255 WatchSource:0}: Error finding container 29482813d34a271ca6c661f440c81bcc73fa6740614e4946c78e3995a90f4255: Status 404 returned error can't find the container with id 29482813d34a271ca6c661f440c81bcc73fa6740614e4946c78e3995a90f4255 Nov 02 20:50:42.618344 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:42.618313 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:50:42.618420 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:42.618377 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:50:42.618420 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:42.618406 4971 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 20:50:42.782314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:42.782282 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:50:42.805070 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:42.805042 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-569554b5f8-8pdkd"] Nov 02 20:50:42.805319 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:42.805294 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-569554b5f8-8pdkd" podUID="de714a26-2a3e-4256-a433-d6c9afc4161c" containerName="manager" containerID="cri-o://c9b062c8935c87c696a35d1ab7caba454da829df8dd19549ebfade073079ff86" gracePeriod=30 Nov 02 20:50:43.085363 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.085004 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569554b5f8-8pdkd" Nov 02 20:50:43.268237 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.268148 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de714a26-2a3e-4256-a433-d6c9afc4161c-kube-api-access-hlgcr\" (UniqueName: \"kubernetes.io/projected/de714a26-2a3e-4256-a433-d6c9afc4161c-kube-api-access-hlgcr\") pod \"de714a26-2a3e-4256-a433-d6c9afc4161c\" (UID: \"de714a26-2a3e-4256-a433-d6c9afc4161c\") " Nov 02 20:50:43.268237 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.268209 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de714a26-2a3e-4256-a433-d6c9afc4161c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/de714a26-2a3e-4256-a433-d6c9afc4161c-webhook-cert\") pod \"de714a26-2a3e-4256-a433-d6c9afc4161c\" (UID: \"de714a26-2a3e-4256-a433-d6c9afc4161c\") " Nov 02 20:50:43.268424 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.268256 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de714a26-2a3e-4256-a433-d6c9afc4161c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de714a26-2a3e-4256-a433-d6c9afc4161c-metrics-certs\") pod \"de714a26-2a3e-4256-a433-d6c9afc4161c\" (UID: \"de714a26-2a3e-4256-a433-d6c9afc4161c\") " Nov 02 20:50:43.271145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.271050 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de714a26-2a3e-4256-a433-d6c9afc4161c-webhook-cert" pod "de714a26-2a3e-4256-a433-d6c9afc4161c" (UID: "de714a26-2a3e-4256-a433-d6c9afc4161c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:50:43.271145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.271065 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de714a26-2a3e-4256-a433-d6c9afc4161c-kube-api-access-hlgcr" pod "de714a26-2a3e-4256-a433-d6c9afc4161c" (UID: "de714a26-2a3e-4256-a433-d6c9afc4161c"). InnerVolumeSpecName "kube-api-access-hlgcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:50:43.271145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.271075 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de714a26-2a3e-4256-a433-d6c9afc4161c-metrics-certs" pod "de714a26-2a3e-4256-a433-d6c9afc4161c" (UID: "de714a26-2a3e-4256-a433-d6c9afc4161c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:50:43.369762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.369726 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlgcr\" (UniqueName: \"kubernetes.io/projected/de714a26-2a3e-4256-a433-d6c9afc4161c-kube-api-access-hlgcr\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:43.369762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.369757 4971 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de714a26-2a3e-4256-a433-d6c9afc4161c-webhook-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:43.369762 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.369767 4971 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de714a26-2a3e-4256-a433-d6c9afc4161c-metrics-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:43.495779 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.495738 4971 generic.go:356] "Generic (PLEG): container finished" podID="de714a26-2a3e-4256-a433-d6c9afc4161c" containerID="c9b062c8935c87c696a35d1ab7caba454da829df8dd19549ebfade073079ff86" exitCode=0 Nov 02 20:50:43.495980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.495792 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569554b5f8-8pdkd" Nov 02 20:50:43.495980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.495824 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569554b5f8-8pdkd" event={"ID":"de714a26-2a3e-4256-a433-d6c9afc4161c","Type":"ContainerDied","Data":"c9b062c8935c87c696a35d1ab7caba454da829df8dd19549ebfade073079ff86"} Nov 02 20:50:43.495980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.495916 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569554b5f8-8pdkd" event={"ID":"de714a26-2a3e-4256-a433-d6c9afc4161c","Type":"ContainerDied","Data":"63438832e7997ce3cdc588ee1a2617f988f54ba76744a804e90364478df5beef"} Nov 02 20:50:43.495980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.495950 4971 scope.go:122] "RemoveContainer" containerID="c9b062c8935c87c696a35d1ab7caba454da829df8dd19549ebfade073079ff86" Nov 02 20:50:43.497807 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.497775 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" event={"ID":"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633","Type":"ContainerStarted","Data":"dfffb3d822b101eacdbfdcf33842170cdb61a69260623800399f79aa5244dd6a"} Nov 02 20:50:43.497928 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.497826 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" event={"ID":"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633","Type":"ContainerStarted","Data":"29482813d34a271ca6c661f440c81bcc73fa6740614e4946c78e3995a90f4255"} Nov 02 20:50:43.506851 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.506836 4971 scope.go:122] "RemoveContainer" containerID="c9b062c8935c87c696a35d1ab7caba454da829df8dd19549ebfade073079ff86" Nov 02 20:50:43.507128 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:50:43.507106 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b062c8935c87c696a35d1ab7caba454da829df8dd19549ebfade073079ff86\": container with ID starting with c9b062c8935c87c696a35d1ab7caba454da829df8dd19549ebfade073079ff86 not found: ID does not exist" containerID="c9b062c8935c87c696a35d1ab7caba454da829df8dd19549ebfade073079ff86" Nov 02 20:50:43.507199 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.507137 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b062c8935c87c696a35d1ab7caba454da829df8dd19549ebfade073079ff86"} err="failed to get container status \"c9b062c8935c87c696a35d1ab7caba454da829df8dd19549ebfade073079ff86\": rpc error: code = NotFound desc = could not find container \"c9b062c8935c87c696a35d1ab7caba454da829df8dd19549ebfade073079ff86\": container with ID starting with c9b062c8935c87c696a35d1ab7caba454da829df8dd19549ebfade073079ff86 not found: ID does not exist" Nov 02 20:50:43.662008 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.661954 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" podStartSLOduration=2.661938256 podStartE2EDuration="2.661938256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:50:43.660520832 +0000 UTC m=+2171.950177359" watchObservedRunningTime="2026-11-02 20:50:43.661938256 +0000 UTC m=+2171.951594784" Nov 02 20:50:43.766286 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.766249 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-569554b5f8-8pdkd"] Nov 02 20:50:43.770332 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.770306 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-569554b5f8-8pdkd"] Nov 02 20:50:43.867210 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:43.867180 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de714a26-2a3e-4256-a433-d6c9afc4161c" path="/var/lib/kubelet/pods/de714a26-2a3e-4256-a433-d6c9afc4161c/volumes" Nov 02 20:50:44.050445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:44.050349 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg"] Nov 02 20:50:44.053180 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:44.053155 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx"] Nov 02 20:50:44.055355 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:44.055340 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-866d5234-n6rcg"] Nov 02 20:50:44.057363 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:44.057345 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4a2cb3d-kg8vx"] Nov 02 20:50:44.194523 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:44.194494 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:44.196026 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:44.195998 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-rvs7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 02 20:50:44.196189 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:44.196055 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" podUID="aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 02 20:50:45.195012 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:45.194978 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-rvs7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 02 20:50:45.195403 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:45.195029 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" podUID="aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 02 20:50:45.261548 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:45.261517 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:50:45.865126 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:45.865075 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b5eee46-301a-46eb-8672-458a4b62a3ec" path="/var/lib/kubelet/pods/3b5eee46-301a-46eb-8672-458a4b62a3ec/volumes" Nov 02 20:50:45.865529 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:45.865515 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42bde7ba-8b52-46dc-98a4-ae6df715ceb0" path="/var/lib/kubelet/pods/42bde7ba-8b52-46dc-98a4-ae6df715ceb0/volumes" Nov 02 20:50:46.195051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:46.194968 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-rvs7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 02 20:50:46.195051 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:46.195024 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" podUID="aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 02 20:50:47.195028 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:47.194993 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-rvs7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 02 20:50:47.195233 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:47.195057 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" podUID="aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 02 20:50:48.194561 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:48.194521 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-rvs7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 02 20:50:48.194756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:48.194575 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" podUID="aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 02 20:50:49.194884 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:49.194851 4971 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-rvs7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 02 20:50:49.195305 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:49.194901 4971 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" podUID="aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 02 20:50:49.939312 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:49.939274 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:50:50.830257 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:50.830218 4971 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:50.830731 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:50.830664 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:50:50.831405 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:50:50.831382 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:51:03.338797 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:51:03.338759 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:51:04.105416 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:51:04.105379 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:51:08.016466 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:51:08.016409 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:51:08.261599 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:51:08.261568 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:51:33.338460 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:51:33.338422 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:51:34.504360 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:51:34.504326 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:51:42.388552 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:51:42.388518 4971 scope.go:122] "RemoveContainer" containerID="851ca3015667062d387db0ab81c090498df8f46930f6ab83cffb1cf0e5a24f75" Nov 02 20:51:42.397915 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:51:42.397896 4971 scope.go:122] "RemoveContainer" containerID="a5845748fb4ef9642240cd69a7f37dc4688643dc386523da250dc602ba0124dd" Nov 02 20:51:42.406348 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:51:42.406328 4971 scope.go:122] "RemoveContainer" containerID="6cdcb09c1f4650a71fa9fe5eef10e93b1ce30f25446aee37ad32f83c95e994ec" Nov 02 20:51:42.414829 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:51:42.414810 4971 scope.go:122] "RemoveContainer" containerID="9ff17db6e47f245e9bf57e5de71e0c370e22f9b4d24d77ed983814016076b1fa" Nov 02 20:51:42.423266 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:51:42.423245 4971 scope.go:122] "RemoveContainer" containerID="21dbc29430303c3cfd7cd2efd396bd078307470a494c77c5b4976a43deb1751c" Nov 02 20:51:44.471445 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:51:44.471407 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:51:44.981483 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:51:44.981452 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:51:45.226134 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:51:45.226069 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:52:03.341168 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:52:03.341131 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:52:33.337235 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:52:33.337198 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:52:34.860298 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:52:34.860249 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:52:42.476205 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:52:42.476119 4971 scope.go:122] "RemoveContainer" containerID="f2b6488b8df54fdec69d66ad0a67c09e48fd1898d74f98c421f5991f6738ed85" Nov 02 20:53:03.340711 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:53:03.340674 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:53:12.197756 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:12.197714 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:12.443571 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:12.443540 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:26.046235 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:26.046203 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6"] Nov 02 20:53:26.095267 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:26.095235 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t"] Nov 02 20:53:26.098774 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:26.098745 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef94b89-94kk6"] Nov 02 20:53:26.100837 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:26.100816 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e3fffe3-vdv8t"] Nov 02 20:53:27.865727 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:27.865692 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be58e326-8b22-4a60-a9fb-0a4546a54ae7" path="/var/lib/kubelet/pods/be58e326-8b22-4a60-a9fb-0a4546a54ae7/volumes" Nov 02 20:53:27.866160 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:27.866146 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64" path="/var/lib/kubelet/pods/ea6d6ca0-4f04-4ee2-b6ad-7b5e4961bc64/volumes" Nov 02 20:53:33.340120 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:53:33.340056 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:53:35.204310 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:35.204266 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:53:37.045645 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:37.045612 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6"] Nov 02 20:53:37.092421 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:37.092386 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn"] Nov 02 20:53:37.094327 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:37.094309 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e18bb6d4-tzlz6"] Nov 02 20:53:37.096236 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:37.096211 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-257e022b-pzmnn"] Nov 02 20:53:37.865915 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:37.865880 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1" path="/var/lib/kubelet/pods/9f508d04-a0a8-4e9c-bc78-cf2e4c6211d1/volumes" Nov 02 20:53:37.866327 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:37.866311 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc54304f-438e-4dd9-8338-16b67bab73b7" path="/var/lib/kubelet/pods/dc54304f-438e-4dd9-8338-16b67bab73b7/volumes" Nov 02 20:53:42.498932 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:42.498893 4971 scope.go:122] "RemoveContainer" containerID="a86d68bbb1fa248dde992c9034e55b680112a6e08b240553b6cf4cad9f5c216a" Nov 02 20:53:42.508507 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:42.508484 4971 scope.go:122] "RemoveContainer" containerID="718491c0ebd7f7a5ea935859d743a1fbb88a1da3399b96b0fc9972aee7c41d41" Nov 02 20:53:42.517508 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:53:42.517483 4971 scope.go:122] "RemoveContainer" containerID="52b7b2d062f17d2ace656ae751e80f1098db3a4228b6132270da2751bc454ae5" Nov 02 20:54:03.339805 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:54:03.339766 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:54:14.045466 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:14.045392 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq"] Nov 02 20:54:14.092274 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:14.092243 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda6b36f-cv9kq"] Nov 02 20:54:15.865826 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:15.865792 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aef9d815-9128-4462-89e7-bca5fe407a55" path="/var/lib/kubelet/pods/aef9d815-9128-4462-89e7-bca5fe407a55/volumes" Nov 02 20:54:24.047357 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:24.047323 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69"] Nov 02 20:54:24.094253 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:24.094218 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d"] Nov 02 20:54:24.096351 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:24.096326 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7f91883-p7k69"] Nov 02 20:54:24.098102 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:24.098066 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01bcb75f-qfb8d"] Nov 02 20:54:25.865280 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:25.865246 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46adb19-9afd-4671-94fe-4a5f6b965dfb" path="/var/lib/kubelet/pods/b46adb19-9afd-4671-94fe-4a5f6b965dfb/volumes" Nov 02 20:54:25.865678 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:25.865652 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e88e7eef-5eb8-4274-ad74-4c691a426260" path="/var/lib/kubelet/pods/e88e7eef-5eb8-4274-ad74-4c691a426260/volumes" Nov 02 20:54:32.263300 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:32.263272 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:54:32.271474 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:32.271455 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:54:33.340352 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:54:33.340316 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:54:35.571546 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:35.571510 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:54:42.548250 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:42.548221 4971 scope.go:122] "RemoveContainer" containerID="648aadaf6c656e2ff3177a43dd6f74d819c85ed1e1be551487cd53422aff3524" Nov 02 20:54:42.557561 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:42.557537 4971 scope.go:122] "RemoveContainer" containerID="cbbea10a947e3dc550dbad7da20d5ad74e9324ec7c1ab63285aed6439512c121" Nov 02 20:54:42.565585 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:42.565567 4971 scope.go:122] "RemoveContainer" containerID="668f338dd66a149e00e582d53f935c5cb6a2d7046394b3620eee79cbadcfa3c9" Nov 02 20:54:42.573606 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:42.573590 4971 scope.go:122] "RemoveContainer" containerID="280f0a44643c9ee9e319a68fe98d0f556484a064c9cac60dabfc20c042f5be9e" Nov 02 20:54:42.581371 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:42.581354 4971 scope.go:122] "RemoveContainer" containerID="9c37f95a70394bdc8104ac83416009a4902148e7f816880dd70d713d28820d45" Nov 02 20:54:42.589200 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:42.589180 4971 scope.go:122] "RemoveContainer" containerID="b47fdc476e7ab895dd5a3e8bc51818234a73ca9bbbd31f8de711f917126a98ba" Nov 02 20:54:42.596957 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:42.596935 4971 scope.go:122] "RemoveContainer" containerID="1a9e7ada215e404d59586e092643db0be90008d29fa8fd469f8c44953e4fba50" Nov 02 20:54:42.604504 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:42.604477 4971 scope.go:122] "RemoveContainer" containerID="f088be4e06509dcf8a7a5f7e25a8834f33d8c14c5c0a10d9c82ce58aa6d4cf86" Nov 02 20:54:42.612294 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:54:42.612273 4971 scope.go:122] "RemoveContainer" containerID="ef03d21af285033422f9a421fab455900c496d3bf1cf66f59a846e85760e204f" Nov 02 20:55:03.338066 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:55:03.338028 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:55:14.842994 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:14.842957 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:55:33.341942 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:55:33.341894 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:55:34.046144 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:34.046112 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx"] Nov 02 20:55:34.092842 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:34.092814 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d6dsx"] Nov 02 20:55:35.866625 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:35.866588 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c85e8926-09a1-4e2f-a110-0e6b17c7a3f2" path="/var/lib/kubelet/pods/c85e8926-09a1-4e2f-a110-0e6b17c7a3f2/volumes" Nov 02 20:55:35.891314 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:35.891286 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:55:42.666978 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:42.666902 4971 scope.go:122] "RemoveContainer" containerID="d2fa78c64cef09118b282d9b6b3cc8bb322fdf3ac620173386425b9372245751" Nov 02 20:55:42.676150 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:42.676129 4971 scope.go:122] "RemoveContainer" containerID="71227b01d4e57e7cf3a521e773269ec6bb7249219b81880f7a00c468de91876e" Nov 02 20:55:42.684563 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:42.684538 4971 scope.go:122] "RemoveContainer" containerID="b16cf3ad34dd26baf821edaa14b9b678629f86c49dea5b36b1625a7dc25e8d62" Nov 02 20:55:42.692360 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:42.692336 4971 scope.go:122] "RemoveContainer" containerID="b54adab38f5f760838845a306e3862fd7b67f3e6a948a0cc3433f168086a5037" Nov 02 20:55:42.699795 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:42.699777 4971 scope.go:122] "RemoveContainer" containerID="27204cd8c802a17b84302877e896f61c9a1b3e9671eb7d54a544522fec25cc6f" Nov 02 20:55:42.707295 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:42.707280 4971 scope.go:122] "RemoveContainer" containerID="533843b7951384c2629b8d6f55e6f7edd9ecc717f6a5e58f4ecde2142b1d861a" Nov 02 20:55:43.635008 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:43.634975 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx"] Nov 02 20:55:43.635250 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:43.635224 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" podUID="eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" containerName="istio-proxy" containerID="cri-o://ba93a618221c6aad8520676c9bbd8018b76b814151a2899698787626a757a537" gracePeriod=30 Nov 02 20:55:43.681938 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:43.681908 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w"] Nov 02 20:55:43.682429 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:43.682164 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" podUID="aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" containerName="istio-proxy" containerID="cri-o://dfffb3d822b101eacdbfdcf33842170cdb61a69260623800399f79aa5244dd6a" gracePeriod=30 Nov 02 20:55:43.683870 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:43.683843 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm"] Nov 02 20:55:43.684053 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:43.684031 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" podUID="69c4c5a9-6da8-4d19-bad1-7fd9c8291349" containerName="istio-proxy" containerID="cri-o://199608e47abe389bdf2ebe26f2bc37519c4e1d893a7d261534a84fc129070747" gracePeriod=30 Nov 02 20:55:43.728750 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:43.728718 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn"] Nov 02 20:55:43.728983 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:43.728957 4971 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" podUID="b58ce9c5-f188-4266-98df-0dcee4207d61" containerName="istio-proxy" containerID="cri-o://b3702c56f0c07814eebe8691d43246d5de11227c2adb4012d41c65f8786e4209" gracePeriod=30 Nov 02 20:55:48.968798 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:48.968762 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:55:49.002897 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.002872 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-envoy\") pod \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " Nov 02 20:55:49.003062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.002958 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-token\" (UniqueName: \"kubernetes.io/projected/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-token\") pod \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " Nov 02 20:55:49.003062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.002994 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-workload-certs\") pod \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " Nov 02 20:55:49.003062 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.003034 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-data\") pod \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " Nov 02 20:55:49.003289 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.003068 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istiod-ca-cert\") pod \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " Nov 02 20:55:49.003289 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.003249 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-credential-socket\") pod \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " Nov 02 20:55:49.003426 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.003363 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-podinfo\") pod \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " Nov 02 20:55:49.003426 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.003398 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-workload-socket\") pod \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " Nov 02 20:55:49.003547 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.003442 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-kube-api-access-k2plr\" (UniqueName: \"kubernetes.io/projected/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-kube-api-access-k2plr\") pod \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\" (UID: \"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8\") " Nov 02 20:55:49.003616 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.003582 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-data" pod "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" (UID: "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.004500 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.003675 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istiod-ca-cert" pod "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" (UID: "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:55:49.004500 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.003891 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.004500 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.003911 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.004500 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.003975 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-workload-certs" pod "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" (UID: "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.004500 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.004139 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-workload-socket" pod "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" (UID: "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.004500 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.004417 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-credential-socket" pod "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" (UID: "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.006047 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.006022 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-envoy" pod "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" (UID: "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.006047 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.006033 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-token" pod "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" (UID: "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:55:49.006209 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.006059 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-kube-api-access-k2plr" pod "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" (UID: "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8"). InnerVolumeSpecName "kube-api-access-k2plr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:55:49.007457 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.007431 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-podinfo" pod "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" (UID: "eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:55:49.027929 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.027903 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:55:49.045738 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.045716 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:55:49.057057 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.057037 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:55:49.104655 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.104631 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-podinfo\") pod \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " Nov 02 20:55:49.104787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.104676 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-workload-socket\") pod \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " Nov 02 20:55:49.104787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.104729 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-workload-socket\") pod \"b58ce9c5-f188-4266-98df-0dcee4207d61\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " Nov 02 20:55:49.104787 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.104765 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-credential-socket\") pod \"b58ce9c5-f188-4266-98df-0dcee4207d61\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " Nov 02 20:55:49.104966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.104795 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-token\" (UniqueName: \"kubernetes.io/projected/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-token\") pod \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " Nov 02 20:55:49.104966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.104836 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istiod-ca-cert\") pod \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " Nov 02 20:55:49.104966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.104868 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-credential-socket\") pod \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " Nov 02 20:55:49.104966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.104922 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-workload-certs\") pod \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " Nov 02 20:55:49.104966 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.104951 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b58ce9c5-f188-4266-98df-0dcee4207d61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b58ce9c5-f188-4266-98df-0dcee4207d61-istiod-ca-cert\") pod \"b58ce9c5-f188-4266-98df-0dcee4207d61\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " Nov 02 20:55:49.105255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105001 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-envoy\") pod \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " Nov 02 20:55:49.105255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.104986 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-workload-socket" pod "b58ce9c5-f188-4266-98df-0dcee4207d61" (UID: "b58ce9c5-f188-4266-98df-0dcee4207d61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.105255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105026 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-workload-socket" pod "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" (UID: "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.105255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105036 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-podinfo\") pod \"b58ce9c5-f188-4266-98df-0dcee4207d61\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " Nov 02 20:55:49.105255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105114 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-workload-certs\") pod \"b58ce9c5-f188-4266-98df-0dcee4207d61\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " Nov 02 20:55:49.105255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105152 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-envoy\") pod \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " Nov 02 20:55:49.105255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105220 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-data\") pod \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " Nov 02 20:55:49.105255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105240 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-credential-socket" pod "69c4c5a9-6da8-4d19-bad1-7fd9c8291349" (UID: "69c4c5a9-6da8-4d19-bad1-7fd9c8291349"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.105255 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105250 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-data\") pod \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " Nov 02 20:55:49.105958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105290 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-workload-certs\") pod \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " Nov 02 20:55:49.105958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105332 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-data\") pod \"b58ce9c5-f188-4266-98df-0dcee4207d61\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " Nov 02 20:55:49.105958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105361 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-podinfo\") pod \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " Nov 02 20:55:49.105958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105392 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-workload-socket\") pod \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " Nov 02 20:55:49.105958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105433 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-envoy\") pod \"b58ce9c5-f188-4266-98df-0dcee4207d61\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " Nov 02 20:55:49.105958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105483 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-kube-api-access-zj2d5\" (UniqueName: \"kubernetes.io/projected/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-kube-api-access-zj2d5\") pod \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " Nov 02 20:55:49.105958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105512 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-kube-api-access-x2sfb\" (UniqueName: \"kubernetes.io/projected/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-kube-api-access-x2sfb\") pod \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " Nov 02 20:55:49.105958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105541 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istiod-ca-cert" pod "69c4c5a9-6da8-4d19-bad1-7fd9c8291349" (UID: "69c4c5a9-6da8-4d19-bad1-7fd9c8291349"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:55:49.105958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105542 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-token\" (UniqueName: \"kubernetes.io/projected/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-token\") pod \"b58ce9c5-f188-4266-98df-0dcee4207d61\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " Nov 02 20:55:49.105958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105587 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-token\" (UniqueName: \"kubernetes.io/projected/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-token\") pod \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\" (UID: \"69c4c5a9-6da8-4d19-bad1-7fd9c8291349\") " Nov 02 20:55:49.105958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105701 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-data" pod "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" (UID: "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.105958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105714 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b58ce9c5-f188-4266-98df-0dcee4207d61-kube-api-access-wv9bs\" (UniqueName: \"kubernetes.io/projected/b58ce9c5-f188-4266-98df-0dcee4207d61-kube-api-access-wv9bs\") pod \"b58ce9c5-f188-4266-98df-0dcee4207d61\" (UID: \"b58ce9c5-f188-4266-98df-0dcee4207d61\") " Nov 02 20:55:49.105958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105754 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istiod-ca-cert\") pod \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " Nov 02 20:55:49.105958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105788 4971 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-credential-socket\") pod \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\" (UID: \"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633\") " Nov 02 20:55:49.105958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105828 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b58ce9c5-f188-4266-98df-0dcee4207d61-istiod-ca-cert" pod "b58ce9c5-f188-4266-98df-0dcee4207d61" (UID: "b58ce9c5-f188-4266-98df-0dcee4207d61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:55:49.105958 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.105946 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-workload-certs" pod "b58ce9c5-f188-4266-98df-0dcee4207d61" (UID: "b58ce9c5-f188-4266-98df-0dcee4207d61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106210 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-data" pod "69c4c5a9-6da8-4d19-bad1-7fd9c8291349" (UID: "69c4c5a9-6da8-4d19-bad1-7fd9c8291349"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106266 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106288 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106302 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106309 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-workload-certs" pod "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" (UID: "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106316 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106353 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106373 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106388 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106402 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106419 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106432 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106446 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106460 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2plr\" (UniqueName: \"kubernetes.io/projected/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-kube-api-access-k2plr\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106475 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b58ce9c5-f188-4266-98df-0dcee4207d61-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106489 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106705 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-workload-certs" pod "69c4c5a9-6da8-4d19-bad1-7fd9c8291349" (UID: "69c4c5a9-6da8-4d19-bad1-7fd9c8291349"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106790 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istiod-ca-cert" pod "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" (UID: "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:55:49.106941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.106913 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-data" pod "b58ce9c5-f188-4266-98df-0dcee4207d61" (UID: "b58ce9c5-f188-4266-98df-0dcee4207d61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.108697 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.107001 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-credential-socket" pod "b58ce9c5-f188-4266-98df-0dcee4207d61" (UID: "b58ce9c5-f188-4266-98df-0dcee4207d61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.108697 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.107564 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-podinfo" pod "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" (UID: "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:55:49.108697 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.107752 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-credential-socket" pod "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" (UID: "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.108697 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.108135 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-workload-socket" pod "69c4c5a9-6da8-4d19-bad1-7fd9c8291349" (UID: "69c4c5a9-6da8-4d19-bad1-7fd9c8291349"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.109104 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.109058 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-podinfo" pod "b58ce9c5-f188-4266-98df-0dcee4207d61" (UID: "b58ce9c5-f188-4266-98df-0dcee4207d61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:55:49.110195 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.110132 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-token" pod "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" (UID: "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:55:49.111479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.111455 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-podinfo" pod "69c4c5a9-6da8-4d19-bad1-7fd9c8291349" (UID: "69c4c5a9-6da8-4d19-bad1-7fd9c8291349"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:55:49.111479 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.111475 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-kube-api-access-zj2d5" pod "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" (UID: "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633"). InnerVolumeSpecName "kube-api-access-zj2d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:55:49.111682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.111593 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b58ce9c5-f188-4266-98df-0dcee4207d61-kube-api-access-wv9bs" pod "b58ce9c5-f188-4266-98df-0dcee4207d61" (UID: "b58ce9c5-f188-4266-98df-0dcee4207d61"). InnerVolumeSpecName "kube-api-access-wv9bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:55:49.111682 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.111616 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-token" pod "69c4c5a9-6da8-4d19-bad1-7fd9c8291349" (UID: "69c4c5a9-6da8-4d19-bad1-7fd9c8291349"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:55:49.111806 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.111716 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-envoy" pod "69c4c5a9-6da8-4d19-bad1-7fd9c8291349" (UID: "69c4c5a9-6da8-4d19-bad1-7fd9c8291349"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.111806 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.111789 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-kube-api-access-x2sfb" pod "69c4c5a9-6da8-4d19-bad1-7fd9c8291349" (UID: "69c4c5a9-6da8-4d19-bad1-7fd9c8291349"). InnerVolumeSpecName "kube-api-access-x2sfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:55:49.111921 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.111823 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-token" pod "b58ce9c5-f188-4266-98df-0dcee4207d61" (UID: "b58ce9c5-f188-4266-98df-0dcee4207d61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:55:49.112177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.112153 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-envoy" pod "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" (UID: "aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.113029 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.113009 4971 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-envoy" pod "b58ce9c5-f188-4266-98df-0dcee4207d61" (UID: "b58ce9c5-f188-4266-98df-0dcee4207d61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:55:49.207444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207369 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj2d5\" (UniqueName: \"kubernetes.io/projected/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-kube-api-access-zj2d5\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207395 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2sfb\" (UniqueName: \"kubernetes.io/projected/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-kube-api-access-x2sfb\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207405 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207415 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207423 4971 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv9bs\" (UniqueName: \"kubernetes.io/projected/b58ce9c5-f188-4266-98df-0dcee4207d61-kube-api-access-wv9bs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207432 4971 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istiod-ca-cert\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207439 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207444 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207447 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207457 4971 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-credential-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207464 4971 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-token\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207472 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207480 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207489 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207498 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207505 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207513 4971 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-workload-certs\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207521 4971 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-data\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207528 4971 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-istio-podinfo\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207536 4971 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c4c5a9-6da8-4d19-bad1-7fd9c8291349-workload-socket\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.207990 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.207544 4971 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b58ce9c5-f188-4266-98df-0dcee4207d61-istio-envoy\") on node \"ip-10-0-140-120.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:55:49.309298 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.309260 4971 generic.go:356] "Generic (PLEG): container finished" podID="aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" containerID="dfffb3d822b101eacdbfdcf33842170cdb61a69260623800399f79aa5244dd6a" exitCode=0 Nov 02 20:55:49.309941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.309299 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" event={"ID":"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633","Type":"ContainerDied","Data":"dfffb3d822b101eacdbfdcf33842170cdb61a69260623800399f79aa5244dd6a"} Nov 02 20:55:49.309941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.309316 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" Nov 02 20:55:49.309941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.309364 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w" event={"ID":"aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633","Type":"ContainerDied","Data":"29482813d34a271ca6c661f440c81bcc73fa6740614e4946c78e3995a90f4255"} Nov 02 20:55:49.309941 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.309397 4971 scope.go:122] "RemoveContainer" containerID="dfffb3d822b101eacdbfdcf33842170cdb61a69260623800399f79aa5244dd6a" Nov 02 20:55:49.311398 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.311370 4971 generic.go:356] "Generic (PLEG): container finished" podID="eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" containerID="ba93a618221c6aad8520676c9bbd8018b76b814151a2899698787626a757a537" exitCode=0 Nov 02 20:55:49.311520 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.311421 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" Nov 02 20:55:49.311520 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.311462 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" event={"ID":"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8","Type":"ContainerDied","Data":"ba93a618221c6aad8520676c9bbd8018b76b814151a2899698787626a757a537"} Nov 02 20:55:49.311631 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.311523 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx" event={"ID":"eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8","Type":"ContainerDied","Data":"ed6283828ec6f35e82eba53f9b564e6ed0ff118584336a4963b2f261e341984e"} Nov 02 20:55:49.315980 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.315921 4971 generic.go:356] "Generic (PLEG): container finished" podID="69c4c5a9-6da8-4d19-bad1-7fd9c8291349" containerID="199608e47abe389bdf2ebe26f2bc37519c4e1d893a7d261534a84fc129070747" exitCode=0 Nov 02 20:55:49.316145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.316033 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" event={"ID":"69c4c5a9-6da8-4d19-bad1-7fd9c8291349","Type":"ContainerDied","Data":"199608e47abe389bdf2ebe26f2bc37519c4e1d893a7d261534a84fc129070747"} Nov 02 20:55:49.316145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.316073 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" event={"ID":"69c4c5a9-6da8-4d19-bad1-7fd9c8291349","Type":"ContainerDied","Data":"0992be9d5bab4fd57a9767904496ddb40fd469282726671bb281b25ab43e6d10"} Nov 02 20:55:49.316475 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.316167 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm" Nov 02 20:55:49.320550 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.320524 4971 generic.go:356] "Generic (PLEG): container finished" podID="b58ce9c5-f188-4266-98df-0dcee4207d61" containerID="b3702c56f0c07814eebe8691d43246d5de11227c2adb4012d41c65f8786e4209" exitCode=0 Nov 02 20:55:49.321201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.320624 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" event={"ID":"b58ce9c5-f188-4266-98df-0dcee4207d61","Type":"ContainerDied","Data":"b3702c56f0c07814eebe8691d43246d5de11227c2adb4012d41c65f8786e4209"} Nov 02 20:55:49.321201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.320665 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" event={"ID":"b58ce9c5-f188-4266-98df-0dcee4207d61","Type":"ContainerDied","Data":"2b613eb42200485ae6083b75e5319e25b5e4bf42094d0bac858b4874bd24a4f7"} Nov 02 20:55:49.321201 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.320730 4971 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn" Nov 02 20:55:49.325490 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.325466 4971 scope.go:122] "RemoveContainer" containerID="dfffb3d822b101eacdbfdcf33842170cdb61a69260623800399f79aa5244dd6a" Nov 02 20:55:49.325826 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:55:49.325799 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfffb3d822b101eacdbfdcf33842170cdb61a69260623800399f79aa5244dd6a\": container with ID starting with dfffb3d822b101eacdbfdcf33842170cdb61a69260623800399f79aa5244dd6a not found: ID does not exist" containerID="dfffb3d822b101eacdbfdcf33842170cdb61a69260623800399f79aa5244dd6a" Nov 02 20:55:49.326478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.325838 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfffb3d822b101eacdbfdcf33842170cdb61a69260623800399f79aa5244dd6a"} err="failed to get container status \"dfffb3d822b101eacdbfdcf33842170cdb61a69260623800399f79aa5244dd6a\": rpc error: code = NotFound desc = could not find container \"dfffb3d822b101eacdbfdcf33842170cdb61a69260623800399f79aa5244dd6a\": container with ID starting with dfffb3d822b101eacdbfdcf33842170cdb61a69260623800399f79aa5244dd6a not found: ID does not exist" Nov 02 20:55:49.326478 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.325864 4971 scope.go:122] "RemoveContainer" containerID="ba93a618221c6aad8520676c9bbd8018b76b814151a2899698787626a757a537" Nov 02 20:55:49.336967 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.336943 4971 scope.go:122] "RemoveContainer" containerID="ba93a618221c6aad8520676c9bbd8018b76b814151a2899698787626a757a537" Nov 02 20:55:49.337269 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:55:49.337251 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba93a618221c6aad8520676c9bbd8018b76b814151a2899698787626a757a537\": container with ID starting with ba93a618221c6aad8520676c9bbd8018b76b814151a2899698787626a757a537 not found: ID does not exist" containerID="ba93a618221c6aad8520676c9bbd8018b76b814151a2899698787626a757a537" Nov 02 20:55:49.337341 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.337276 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba93a618221c6aad8520676c9bbd8018b76b814151a2899698787626a757a537"} err="failed to get container status \"ba93a618221c6aad8520676c9bbd8018b76b814151a2899698787626a757a537\": rpc error: code = NotFound desc = could not find container \"ba93a618221c6aad8520676c9bbd8018b76b814151a2899698787626a757a537\": container with ID starting with ba93a618221c6aad8520676c9bbd8018b76b814151a2899698787626a757a537 not found: ID does not exist" Nov 02 20:55:49.337341 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.337291 4971 scope.go:122] "RemoveContainer" containerID="199608e47abe389bdf2ebe26f2bc37519c4e1d893a7d261534a84fc129070747" Nov 02 20:55:49.345738 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.345721 4971 scope.go:122] "RemoveContainer" containerID="199608e47abe389bdf2ebe26f2bc37519c4e1d893a7d261534a84fc129070747" Nov 02 20:55:49.345998 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:55:49.345974 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199608e47abe389bdf2ebe26f2bc37519c4e1d893a7d261534a84fc129070747\": container with ID starting with 199608e47abe389bdf2ebe26f2bc37519c4e1d893a7d261534a84fc129070747 not found: ID does not exist" containerID="199608e47abe389bdf2ebe26f2bc37519c4e1d893a7d261534a84fc129070747" Nov 02 20:55:49.346127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.346004 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199608e47abe389bdf2ebe26f2bc37519c4e1d893a7d261534a84fc129070747"} err="failed to get container status \"199608e47abe389bdf2ebe26f2bc37519c4e1d893a7d261534a84fc129070747\": rpc error: code = NotFound desc = could not find container \"199608e47abe389bdf2ebe26f2bc37519c4e1d893a7d261534a84fc129070747\": container with ID starting with 199608e47abe389bdf2ebe26f2bc37519c4e1d893a7d261534a84fc129070747 not found: ID does not exist" Nov 02 20:55:49.346127 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.346026 4971 scope.go:122] "RemoveContainer" containerID="b3702c56f0c07814eebe8691d43246d5de11227c2adb4012d41c65f8786e4209" Nov 02 20:55:49.354258 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.354239 4971 scope.go:122] "RemoveContainer" containerID="b3702c56f0c07814eebe8691d43246d5de11227c2adb4012d41c65f8786e4209" Nov 02 20:55:49.354483 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:55:49.354466 4971 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3702c56f0c07814eebe8691d43246d5de11227c2adb4012d41c65f8786e4209\": container with ID starting with b3702c56f0c07814eebe8691d43246d5de11227c2adb4012d41c65f8786e4209 not found: ID does not exist" containerID="b3702c56f0c07814eebe8691d43246d5de11227c2adb4012d41c65f8786e4209" Nov 02 20:55:49.354541 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.354492 4971 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3702c56f0c07814eebe8691d43246d5de11227c2adb4012d41c65f8786e4209"} err="failed to get container status \"b3702c56f0c07814eebe8691d43246d5de11227c2adb4012d41c65f8786e4209\": rpc error: code = NotFound desc = could not find container \"b3702c56f0c07814eebe8691d43246d5de11227c2adb4012d41c65f8786e4209\": container with ID starting with b3702c56f0c07814eebe8691d43246d5de11227c2adb4012d41c65f8786e4209 not found: ID does not exist" Nov 02 20:55:49.537779 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.530675 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn"] Nov 02 20:55:49.540420 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.540394 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-l2grn"] Nov 02 20:55:49.645906 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.645874 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx"] Nov 02 20:55:49.651130 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.651101 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-t9vhx"] Nov 02 20:55:49.755039 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.755010 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w"] Nov 02 20:55:49.758773 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.758745 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-rvs7w"] Nov 02 20:55:49.865264 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.865238 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" path="/var/lib/kubelet/pods/aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633/volumes" Nov 02 20:55:49.865715 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.865699 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b58ce9c5-f188-4266-98df-0dcee4207d61" path="/var/lib/kubelet/pods/b58ce9c5-f188-4266-98df-0dcee4207d61/volumes" Nov 02 20:55:49.866145 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.866131 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" path="/var/lib/kubelet/pods/eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8/volumes" Nov 02 20:55:49.866711 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.866696 4971 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm"] Nov 02 20:55:49.866761 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:49.866726 4971 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-p96bm"] Nov 02 20:55:51.864909 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:55:51.864872 4971 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69c4c5a9-6da8-4d19-bad1-7fd9c8291349" path="/var/lib/kubelet/pods/69c4c5a9-6da8-4d19-bad1-7fd9c8291349/volumes" Nov 02 20:56:03.339598 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:56:03.339558 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:56:10.910184 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:56:10.910147 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:56:12.173969 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:56:12.173936 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:56:33.337684 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:56:33.337645 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:56:36.223372 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:56:36.223334 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:56:42.744771 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:56:42.744738 4971 scope.go:122] "RemoveContainer" containerID="3a467d114173b3408abd710d84163bea283e824744b705ee208fed2ec46a80c8" Nov 02 20:57:03.340941 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:57:03.340903 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:57:33.337844 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:57:33.337765 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:57:36.538769 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:57:36.538735 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:58:03.338232 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:58:03.338185 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:58:33.337515 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:58:33.337476 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:58:36.868269 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:58:36.868220 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 20:59:03.338530 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:59:03.338452 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:59:32.297926 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:59:32.297897 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:59:32.306670 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:59:32.306650 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 20:59:33.337463 ip-10-0-140-120 kubenswrapper[4971]: E1102 20:59:33.337425 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 20:59:37.197177 ip-10-0-140-120 kubenswrapper[4971]: I1102 20:59:37.197125 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 21:00:03.338445 ip-10-0-140-120 kubenswrapper[4971]: E1102 21:00:03.338410 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 21:00:33.339245 ip-10-0-140-120 kubenswrapper[4971]: E1102 21:00:33.339167 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 21:00:37.589812 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:00:37.589771 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 21:00:39.315459 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:00:39.315424 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 21:01:03.341111 ip-10-0-140-120 kubenswrapper[4971]: E1102 21:01:03.341049 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 21:01:33.339308 ip-10-0-140-120 kubenswrapper[4971]: E1102 21:01:33.339267 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 21:01:37.875982 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:01:37.875943 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 21:01:52.387382 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:01:52.387302 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 21:01:53.445052 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:01:53.445015 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 21:02:03.340759 ip-10-0-140-120 kubenswrapper[4971]: E1102 21:02:03.340715 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 21:02:33.016942 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:02:33.016913 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85d44d9c5c-jglfz_585cb390-ad62-4295-8f12-04179043e96c/authorino/0.log" Nov 02 21:02:33.338385 ip-10-0-140-120 kubenswrapper[4971]: E1102 21:02:33.338297 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 21:02:38.215655 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:02:38.215618 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 21:02:54.348165 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:02:54.348132 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-8ggqj_b2c5de9e-e657-4c2b-a5bc-7807bf957e61/manager/0.log" Nov 02 21:02:55.676593 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:02:55.676557 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dd6xp_f80e1293-d112-436c-bb77-31eea6da6fe2/manager/0.log" Nov 02 21:02:57.220454 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:02:57.220421 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-rqdnx_57bf0b41-d2ed-4d88-9ecf-8d46a2b2639c/postgres/0.log" Nov 02 21:02:58.703915 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:02:58.703879 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85d44d9c5c-jglfz_585cb390-ad62-4295-8f12-04179043e96c/authorino/0.log" Nov 02 21:02:59.122664 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:02:59.122613 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-wztv4_bcf7c59a-ef20-4eea-893d-64595f68ff8d/manager/0.log" Nov 02 21:02:59.354945 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:02:59.354911 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-8527l_8a6c5615-a45d-40bf-a881-b1f7c32c0ffd/manager/0.log" Nov 02 21:03:01.676830 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:01.676795 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-65844c896fdfxx4_232e2115-ae5c-4105-8b4f-f74061831f82/istio-proxy/0.log" Nov 02 21:03:02.991516 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:02.991482 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn_37ff3ea1-40c2-4be8-82bf-af4ae749ebc0/istio-proxy/0.log" Nov 02 21:03:03.338553 ip-10-0-140-120 kubenswrapper[4971]: E1102 21:03:03.338406 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 21:03:03.668813 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:03.668783 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f6f5bd44c-4tksg_cb6295a6-f31e-48a3-b4df-7d872a961161/router/0.log" Nov 02 21:03:04.710220 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:04.710185 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm_77b57b66-cb67-4bc2-ac17-c04b63c264d2/main/0.log" Nov 02 21:03:05.171892 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:05.171861 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-4wj6r_1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a/main/0.log" Nov 02 21:03:05.404027 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:05.403978 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-hc8gn_f0316745-a138-4b26-9f2a-a9f5ffd9d97c/main/0.log" Nov 02 21:03:06.267416 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:06.267382 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt_5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361/main/0.log" Nov 02 21:03:07.188298 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:07.188269 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85d44d9c5c-jglfz_585cb390-ad62-4295-8f12-04179043e96c/authorino/0.log" Nov 02 21:03:17.032447 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:17.032371 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-8ggqj_b2c5de9e-e657-4c2b-a5bc-7807bf957e61/manager/0.log" Nov 02 21:03:18.174984 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:18.174952 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-dd6xp_f80e1293-d112-436c-bb77-31eea6da6fe2/manager/0.log" Nov 02 21:03:30.013306 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:30.013272 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85d44d9c5c-jglfz_585cb390-ad62-4295-8f12-04179043e96c/authorino/0.log" Nov 02 21:03:30.415641 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:30.415610 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-wztv4_bcf7c59a-ef20-4eea-893d-64595f68ff8d/manager/0.log" Nov 02 21:03:30.647936 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:30.647907 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-8527l_8a6c5615-a45d-40bf-a881-b1f7c32c0ffd/manager/0.log" Nov 02 21:03:32.376499 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:32.376465 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-65844c896fdfxx4_232e2115-ae5c-4105-8b4f-f74061831f82/istio-proxy/0.log" Nov 02 21:03:33.338043 ip-10-0-140-120 kubenswrapper[4971]: E1102 21:03:33.338009 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 21:03:33.644571 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:33.644544 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-744fc677d7-4r9nn_37ff3ea1-40c2-4be8-82bf-af4ae749ebc0/istio-proxy/0.log" Nov 02 21:03:34.322506 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:34.322465 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f6f5bd44c-4tksg_cb6295a6-f31e-48a3-b4df-7d872a961161/router/0.log" Nov 02 21:03:34.910585 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:34.910547 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-mm6xm_77b57b66-cb67-4bc2-ac17-c04b63c264d2/main/0.log" Nov 02 21:03:35.365108 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:35.365056 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-4wj6r_1b2ab6f3-51c8-47b6-bbd8-d0fde651c16a/main/0.log" Nov 02 21:03:35.444912 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:35.444881 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 21:03:35.582386 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:35.582356 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-hc8gn_f0316745-a138-4b26-9f2a-a9f5ffd9d97c/main/0.log" Nov 02 21:03:36.493912 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:36.493880 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-h92wt_5f4bc85d-53a1-44c4-a8bd-88b7b5b6e361/main/0.log" Nov 02 21:03:38.540838 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:03:38.540804 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 21:04:03.340718 ip-10-0-140-120 kubenswrapper[4971]: E1102 21:04:03.340679 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 21:04:32.333072 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:04:32.333040 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 21:04:32.342041 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:04:32.342021 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2kxv2_a6fea55b-74c2-40b1-9952-cd767e448317/console-operator/2.log" Nov 02 21:04:33.338330 ip-10-0-140-120 kubenswrapper[4971]: E1102 21:04:33.338290 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 21:04:38.887429 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:04:38.887397 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:03.341770 ip-10-0-140-120 kubenswrapper[4971]: E1102 21:05:03.341679 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 21:05:10.645043 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.645006 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p7vqs/must-gather-xs4q5"] Nov 02 21:05:10.645597 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.645461 4971 cpu_manager.go:414] "containerMap: removing container" podUID="de714a26-2a3e-4256-a433-d6c9afc4161c" containerName="manager" Nov 02 21:05:10.645597 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.645476 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de714a26-2a3e-4256-a433-d6c9afc4161c" containerName="manager" Nov 02 21:05:10.645597 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.645489 4971 cpu_manager.go:414] "containerMap: removing container" podUID="69c4c5a9-6da8-4d19-bad1-7fd9c8291349" containerName="istio-proxy" Nov 02 21:05:10.645597 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.645494 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69c4c5a9-6da8-4d19-bad1-7fd9c8291349" containerName="istio-proxy" Nov 02 21:05:10.645597 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.645508 4971 cpu_manager.go:414] "containerMap: removing container" podUID="b58ce9c5-f188-4266-98df-0dcee4207d61" containerName="istio-proxy" Nov 02 21:05:10.645597 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.645514 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b58ce9c5-f188-4266-98df-0dcee4207d61" containerName="istio-proxy" Nov 02 21:05:10.645597 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.645529 4971 cpu_manager.go:414] "containerMap: removing container" podUID="aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" containerName="istio-proxy" Nov 02 21:05:10.645597 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.645534 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" containerName="istio-proxy" Nov 02 21:05:10.645597 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.645546 4971 cpu_manager.go:414] "containerMap: removing container" podUID="eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" containerName="istio-proxy" Nov 02 21:05:10.645597 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.645551 4971 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" containerName="istio-proxy" Nov 02 21:05:10.645928 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.645619 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="b58ce9c5-f188-4266-98df-0dcee4207d61" containerName="istio-proxy" Nov 02 21:05:10.645928 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.645631 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff2ae3c-c1ce-4db3-8ac7-1ee6ac803633" containerName="istio-proxy" Nov 02 21:05:10.645928 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.645640 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb306a7c-bde9-43de-9b5a-bf0cc76a6ec8" containerName="istio-proxy" Nov 02 21:05:10.645928 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.645649 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="de714a26-2a3e-4256-a433-d6c9afc4161c" containerName="manager" Nov 02 21:05:10.645928 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.645653 4971 memory_manager.go:365] "RemoveStaleState removing state" podUID="69c4c5a9-6da8-4d19-bad1-7fd9c8291349" containerName="istio-proxy" Nov 02 21:05:10.649071 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.649045 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p7vqs/must-gather-xs4q5" Nov 02 21:05:10.698436 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.698410 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p7vqs\"/\"kube-root-ca.crt\"" Nov 02 21:05:10.745362 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.745334 4971 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p7vqs\"/\"openshift-service-ca.crt\"" Nov 02 21:05:10.745498 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.745338 4971 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p7vqs\"/\"default-dockercfg-rkz88\"" Nov 02 21:05:10.747377 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.747353 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p7vqs/must-gather-xs4q5"] Nov 02 21:05:10.749715 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.749695 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd8c87ac-2b96-49d7-9dbd-41f94b9868d3-must-gather-output\") pod \"must-gather-xs4q5\" (UID: \"cd8c87ac-2b96-49d7-9dbd-41f94b9868d3\") " pod="openshift-must-gather-p7vqs/must-gather-xs4q5" Nov 02 21:05:10.749804 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.749789 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wnfb\" (UniqueName: \"kubernetes.io/projected/cd8c87ac-2b96-49d7-9dbd-41f94b9868d3-kube-api-access-6wnfb\") pod \"must-gather-xs4q5\" (UID: \"cd8c87ac-2b96-49d7-9dbd-41f94b9868d3\") " pod="openshift-must-gather-p7vqs/must-gather-xs4q5" Nov 02 21:05:10.850161 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.850137 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wnfb\" (UniqueName: \"kubernetes.io/projected/cd8c87ac-2b96-49d7-9dbd-41f94b9868d3-kube-api-access-6wnfb\") pod \"must-gather-xs4q5\" (UID: \"cd8c87ac-2b96-49d7-9dbd-41f94b9868d3\") " pod="openshift-must-gather-p7vqs/must-gather-xs4q5" Nov 02 21:05:10.850283 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.850246 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd8c87ac-2b96-49d7-9dbd-41f94b9868d3-must-gather-output\") pod \"must-gather-xs4q5\" (UID: \"cd8c87ac-2b96-49d7-9dbd-41f94b9868d3\") " pod="openshift-must-gather-p7vqs/must-gather-xs4q5" Nov 02 21:05:10.850582 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.850562 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd8c87ac-2b96-49d7-9dbd-41f94b9868d3-must-gather-output\") pod \"must-gather-xs4q5\" (UID: \"cd8c87ac-2b96-49d7-9dbd-41f94b9868d3\") " pod="openshift-must-gather-p7vqs/must-gather-xs4q5" Nov 02 21:05:10.854530 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.854505 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:10.908460 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.908410 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wnfb\" (UniqueName: \"kubernetes.io/projected/cd8c87ac-2b96-49d7-9dbd-41f94b9868d3-kube-api-access-6wnfb\") pod \"must-gather-xs4q5\" (UID: \"cd8c87ac-2b96-49d7-9dbd-41f94b9868d3\") " pod="openshift-must-gather-p7vqs/must-gather-xs4q5" Nov 02 21:05:10.962988 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:10.962970 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p7vqs/must-gather-xs4q5" Nov 02 21:05:11.102363 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:11.102317 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:11.336645 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:11.336618 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p7vqs/must-gather-xs4q5"] Nov 02 21:05:11.338671 ip-10-0-140-120 kubenswrapper[4971]: W1102 21:05:11.338641 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd8c87ac_2b96_49d7_9dbd_41f94b9868d3.slice/crio-784c9053b8f072481f5c51d4aaa1061c1eba0969e9e2a5b1b06423e94025ddb9 WatchSource:0}: Error finding container 784c9053b8f072481f5c51d4aaa1061c1eba0969e9e2a5b1b06423e94025ddb9: Status 404 returned error can't find the container with id 784c9053b8f072481f5c51d4aaa1061c1eba0969e9e2a5b1b06423e94025ddb9 Nov 02 21:05:11.340742 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:11.340723 4971 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:05:12.063366 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:12.063334 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p7vqs/must-gather-xs4q5" event={"ID":"cd8c87ac-2b96-49d7-9dbd-41f94b9868d3","Type":"ContainerStarted","Data":"784c9053b8f072481f5c51d4aaa1061c1eba0969e9e2a5b1b06423e94025ddb9"} Nov 02 21:05:13.070889 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:13.070802 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p7vqs/must-gather-xs4q5" event={"ID":"cd8c87ac-2b96-49d7-9dbd-41f94b9868d3","Type":"ContainerStarted","Data":"f0687fe05bf6f7aa9cc769348556635e03cdebad84ca93a0f412ff12ba7fd321"} Nov 02 21:05:13.070889 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:13.070855 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p7vqs/must-gather-xs4q5" event={"ID":"cd8c87ac-2b96-49d7-9dbd-41f94b9868d3","Type":"ContainerStarted","Data":"a8e42fb5cfd2d40ac3303fa7a9194637e09373d7141b7d17a90fc02055b775cb"} Nov 02 21:05:13.179940 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:13.179879 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-p7vqs/must-gather-xs4q5" podStartSLOduration=1.887141204 podStartE2EDuration="3.179866916s" totalImagesPullingTime="1.292725712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:05:13.179543017 +0000 UTC m=+3041.469199544" watchObservedRunningTime="2026-11-02 21:05:13.179866916 +0000 UTC m=+3041.469523443" Nov 02 21:05:19.294968 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:19.294930 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jjtbq_afe07c4f-7282-4239-9761-c61ef4153cd9/global-pull-secret-syncer/0.log" Nov 02 21:05:21.157979 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:21.157945 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5bk6w_aef015e0-b0a9-4c9f-8140-dd74d8265e64/connection-checker/0.log" Nov 02 21:05:21.569155 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:21.568583 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7fd7l_22cd77ec-a3f8-4e80-8ffb-def069d70e34/connection-checker/0.log" Nov 02 21:05:22.005651 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:22.005598 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7shr8_5013aa7f-b2fd-4df0-b5e3-3478bef510ee/connection-checker/0.log" Nov 02 21:05:22.501587 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:22.501552 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" Nov 02 21:05:24.436593 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:24.436556 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-120.us-west-2.compute.internal_b918fb29af61c47ab1271b4e381400d1/haproxy/0.log" Nov 02 21:05:28.860328 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:28.860296 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85d44d9c5c-jglfz_585cb390-ad62-4295-8f12-04179043e96c/authorino/0.log" Nov 02 21:05:29.169155 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:29.168851 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-wztv4_bcf7c59a-ef20-4eea-893d-64595f68ff8d/manager/0.log" Nov 02 21:05:29.564392 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:29.564307 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-8527l_8a6c5615-a45d-40bf-a881-b1f7c32c0ffd/manager/0.log" Nov 02 21:05:29.872679 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:29.872646 4971 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5"] Nov 02 21:05:29.878833 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:29.878484 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:29.964634 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:29.964586 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5"] Nov 02 21:05:29.968521 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:29.968486 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/762e202e-f3e3-41ea-91a8-1e79575f53ef-lib-modules\") pod \"perf-node-gather-daemonset-2qpm5\" (UID: \"762e202e-f3e3-41ea-91a8-1e79575f53ef\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:29.968693 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:29.968553 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkpkj\" (UniqueName: \"kubernetes.io/projected/762e202e-f3e3-41ea-91a8-1e79575f53ef-kube-api-access-mkpkj\") pod \"perf-node-gather-daemonset-2qpm5\" (UID: \"762e202e-f3e3-41ea-91a8-1e79575f53ef\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:29.968693 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:29.968679 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/762e202e-f3e3-41ea-91a8-1e79575f53ef-podres\") pod \"perf-node-gather-daemonset-2qpm5\" (UID: \"762e202e-f3e3-41ea-91a8-1e79575f53ef\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:29.968839 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:29.968727 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/762e202e-f3e3-41ea-91a8-1e79575f53ef-proc\") pod \"perf-node-gather-daemonset-2qpm5\" (UID: \"762e202e-f3e3-41ea-91a8-1e79575f53ef\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:29.968839 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:29.968756 4971 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/762e202e-f3e3-41ea-91a8-1e79575f53ef-sys\") pod \"perf-node-gather-daemonset-2qpm5\" (UID: \"762e202e-f3e3-41ea-91a8-1e79575f53ef\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:30.069837 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:30.069799 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/762e202e-f3e3-41ea-91a8-1e79575f53ef-proc\") pod \"perf-node-gather-daemonset-2qpm5\" (UID: \"762e202e-f3e3-41ea-91a8-1e79575f53ef\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:30.069837 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:30.069840 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/762e202e-f3e3-41ea-91a8-1e79575f53ef-sys\") pod \"perf-node-gather-daemonset-2qpm5\" (UID: \"762e202e-f3e3-41ea-91a8-1e79575f53ef\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:30.070057 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:30.069923 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/762e202e-f3e3-41ea-91a8-1e79575f53ef-proc\") pod \"perf-node-gather-daemonset-2qpm5\" (UID: \"762e202e-f3e3-41ea-91a8-1e79575f53ef\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:30.070057 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:30.069928 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/762e202e-f3e3-41ea-91a8-1e79575f53ef-lib-modules\") pod \"perf-node-gather-daemonset-2qpm5\" (UID: \"762e202e-f3e3-41ea-91a8-1e79575f53ef\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:30.070057 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:30.069988 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/762e202e-f3e3-41ea-91a8-1e79575f53ef-lib-modules\") pod \"perf-node-gather-daemonset-2qpm5\" (UID: \"762e202e-f3e3-41ea-91a8-1e79575f53ef\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:30.070057 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:30.070003 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/762e202e-f3e3-41ea-91a8-1e79575f53ef-sys\") pod \"perf-node-gather-daemonset-2qpm5\" (UID: \"762e202e-f3e3-41ea-91a8-1e79575f53ef\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:30.070057 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:30.070013 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkpkj\" (UniqueName: \"kubernetes.io/projected/762e202e-f3e3-41ea-91a8-1e79575f53ef-kube-api-access-mkpkj\") pod \"perf-node-gather-daemonset-2qpm5\" (UID: \"762e202e-f3e3-41ea-91a8-1e79575f53ef\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:30.070057 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:30.070058 4971 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/762e202e-f3e3-41ea-91a8-1e79575f53ef-podres\") pod \"perf-node-gather-daemonset-2qpm5\" (UID: \"762e202e-f3e3-41ea-91a8-1e79575f53ef\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:30.070358 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:30.070169 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/762e202e-f3e3-41ea-91a8-1e79575f53ef-podres\") pod \"perf-node-gather-daemonset-2qpm5\" (UID: \"762e202e-f3e3-41ea-91a8-1e79575f53ef\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:30.128450 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:30.128387 4971 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkpkj\" (UniqueName: \"kubernetes.io/projected/762e202e-f3e3-41ea-91a8-1e79575f53ef-kube-api-access-mkpkj\") pod \"perf-node-gather-daemonset-2qpm5\" (UID: \"762e202e-f3e3-41ea-91a8-1e79575f53ef\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:30.191056 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:30.191020 4971 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:30.562348 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:30.562322 4971 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5"] Nov 02 21:05:30.564287 ip-10-0-140-120 kubenswrapper[4971]: W1102 21:05:30.564237 4971 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod762e202e_f3e3_41ea_91a8_1e79575f53ef.slice/crio-ed0185a5c6ecef1b7bcd1d8b74940f53276d9c537efb45abb5719bbf4a3db027 WatchSource:0}: Error finding container ed0185a5c6ecef1b7bcd1d8b74940f53276d9c537efb45abb5719bbf4a3db027: Status 404 returned error can't find the container with id ed0185a5c6ecef1b7bcd1d8b74940f53276d9c537efb45abb5719bbf4a3db027 Nov 02 21:05:31.204122 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:31.204055 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" event={"ID":"762e202e-f3e3-41ea-91a8-1e79575f53ef","Type":"ContainerStarted","Data":"0d7b619da4fc2c96645f70a376033c9b92fb25de61fdae69a7afd8125294db4e"} Nov 02 21:05:31.204529 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:31.204150 4971 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" event={"ID":"762e202e-f3e3-41ea-91a8-1e79575f53ef","Type":"ContainerStarted","Data":"ed0185a5c6ecef1b7bcd1d8b74940f53276d9c537efb45abb5719bbf4a3db027"} Nov 02 21:05:31.204529 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:31.204281 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:33.338324 ip-10-0-140-120 kubenswrapper[4971]: E1102 21:05:33.338285 4971 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l4q8k" containerName="extractor" Nov 02 21:05:36.646506 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:36.646474 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-ndzpm_6fea26b0-2cef-4683-a20f-620ef0880bfc/cluster-monitoring-operator/0.log" Nov 02 21:05:37.222130 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:37.222100 4971 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" Nov 02 21:05:37.332257 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:37.332204 4971 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-2qpm5" podStartSLOduration=8.332190246 podStartE2EDuration="8.332190246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:05:31.313119867 +0000 UTC m=+3059.602776394" watchObservedRunningTime="2026-11-02 21:05:37.332190246 +0000 UTC m=+3065.621846818" Nov 02 21:05:39.311003 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:39.310972 4971 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jbk5q_b51723af-6a33-4535-91c0-905ce2a51b24/konnectivity-agent/0.log" limit=1024 Nov 02 21:05:40.189924 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:40.189884 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-869gg_cd2244bb-1e62-47be-9e7e-d80f39e2a525/node-exporter/0.log" Nov 02 21:05:40.581471 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:40.581395 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-869gg_cd2244bb-1e62-47be-9e7e-d80f39e2a525/kube-rbac-proxy/0.log" Nov 02 21:05:40.973349 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:40.973298 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-869gg_cd2244bb-1e62-47be-9e7e-d80f39e2a525/init-textfile/0.log" Nov 02 21:05:47.348756 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:47.348728 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-b94zk_1f6f9164-5e01-4400-a606-adf3c5016a61/prometheus-operator-admission-webhook/0.log" Nov 02 21:05:52.762750 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:52.762722 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-65844c896fdfxx4_232e2115-ae5c-4105-8b4f-f74061831f82/istio-proxy/0.log" Nov 02 21:05:53.655577 ip-10-0-140-120 kubenswrapper[4971]: I1102 21:05:53.655546 4971 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-89j9l_9d5e73f9-907c-40b1-a596-b079ded3abfb/networking-console-plugin/0.log"